Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/latest/pull_cdc_integration_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff Agent pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-36k74-z0627 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ce6720c7d45c73879948a67e0703af02fd4c7858" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-36k74" name: "pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" First time build. Skipping changelog. [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} _=/usr/bin/printenv TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-36k74 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=84db20ae-e405-4003-8561-5e723671c88d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Debug info JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=3 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff pingcap_tiflow_pull_cdc_integration_test_3827-36k74 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-ff467ea3dec646dcb34a434b6d5ddeaf65f47c579492826166e1422a0f0cf5b0 + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.23.3' GODEBUG='' GOTELEMETRY='local' GOTELEMETRYDIR='/home/jenkins/.config/go/telemetry' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2601811398=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff bash' debug command: kubectl -n jenkins-tiflow exec -ti pingcap-tiflow-pull-cdc-integration-test-3827-36k74-z0627-hcfff bash [Pipeline] container [Pipeline] { [Pipeline] sh > git rev-list --no-walk 3e31ec89ba643faa364a78dbc7f062bb55f9c486 # timeout=10 + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9508 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: ec4a523fb8f76924 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 27 IN A 20.27.177.113 github.com. 27 IN A 20.205.243.166 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Fri Apr 04 07:17:34 UTC 2025 ;; MSG SIZE rcvd: 103 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12118/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12118/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/12118/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/12118/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [go.mod, go.sum] [Pipeline] echo diff file not matched: go.mod [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-master) 192245760 bytes in 0.70 secs (274151092 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.43.5 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/.git/ .git HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) POST git-upload-pack (656 bytes) POST git-upload-pack (973 bytes) From https://github.com/pingcap/tiflow = [up to date] master -> origin/master * [new ref] refs/pull/12118/head -> origin/pr/12118/head HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) ๐Ÿšง Checkouting to base SHA:4c631d5951854ac91765b729fe3849d8e8d5ed8a... HEAD is now at 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 4c631d5951854ac91765b729fe3849d8e8d5ed8a 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) 9ed3d76c5 consumer(ticdc): adjust consumer log level (#12092) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 4c631d5951854ac91765b729fe3849d8e8d5ed8a ... Updating 4c631d595..a306f4894 Fast-forward go.mod | 5 ++--- go.sum | 8 ++------ 2 files changed, 4 insertions(+), 9 deletions(-) ๐Ÿงพ Pre-merged result: a306f48949e037965a7c1963f500f7e0abbfba51 a306f4894 dm: upgrade go-mysql to v1.12.0 4c631d595 kafka(ticdc): ddl sink close the underline sink if send ddl or checkpoint failed and refactor the kafka ddl sink (#12112) 3cf60079e codec(ticdc): modify DDL type support about debezium protocol (#12113) โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-4c631d5-a306f48 already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/third_party_download [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] echo computeBranchFromPR component: tidb, prTargetBranch: master, prTitle: dm: upgrade go-mysql to v1.12.0, trunkBranch: master [Pipeline] echo computeBranchFromPR component: pd, prTargetBranch: master, prTitle: dm: upgrade go-mysql to v1.12.0, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tikv, prTargetBranch: master, prTitle: dm: upgrade go-mysql to v1.12.0, trunkBranch: master [Pipeline] echo computeBranchFromPR component: tiflash, prTargetBranch: master, prTitle: dm: upgrade go-mysql to v1.12.0, trunkBranch: master [Pipeline] retry [Pipeline] { [Pipeline] sh + export TIDB_BRANCH=master + TIDB_BRANCH=master + export PD_BRANCH=master + PD_BRANCH=master + export TIKV_BRANCH=master + TIKV_BRANCH=master + export TIFLASH_BRANCH=master + TIFLASH_BRANCH=master + cd ../tiflow + ./scripts/download-integration-test-binaries.sh master Downloading binaries... >>> Downloading tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz 2025-04-04 15:18:06 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/d9fb3a53376157be49b5a7941e6e689a33c81eb0/centos7/tidb-server.tar.gz [620275338/620275338] -> "tmp/tidb-server.tar.gz" [1] >>> Downloading pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz 2025-04-04 15:18:24 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/9d6bbb8529d9d2acb9cf324a65e94ece465f6346/centos7/pd-server.tar.gz [228444265/228444265] -> "tmp/pd-server.tar.gz" [1] >>> Downloading tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz 2025-04-04 15:18:50 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/3eb3439fd88f0db9de06ef92e845e80b32c9b49a/centos7/tikv-server.tar.gz [817400808/817400808] -> "tmp/tikv-server.tar.gz" [1] >>> Downloading tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz 2025-04-04 15:19:19 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/master/8ecb79fcf30098e6786f4d0440ead2a879a3ad4a/centos7/tiflash.tar.gz [553235170/553235170] -> "tmp/tiflash.tar.gz" [1] >>> Downloading minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2025-04-04 15:19:25 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> Downloading etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2025-04-04 15:19:27 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> Downloading schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2025-04-04 15:19:38 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] >>> Downloading go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2025-04-04 15:19:41 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> Downloading jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2025-04-04 15:19:41 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] Download SUCCESS + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:19 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:19 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:19 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:19 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + make check_third_party_binary go: downloading go1.23.4 (linux/amd64) /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-region-bench ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/pd-ut ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ../tiflow/bin/xprog ./bin/ + ls -alh ./bin total 2.0G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:19 . drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:19 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:19 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:19 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Dec 18 2023 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 3 12:33 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Dec 18 2023 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 3 12:33 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 3 12:33 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 3 11:49 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 3 12:34 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 41M Apr 3 15:30 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 3 15:29 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 3 15:30 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:29 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 3 15:29 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 3 15:30 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 3 15:30 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 3 15:30 stores-dump -rwxr-xr-x 1 jenkins jenkins 241M Apr 3 19:14 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 3 12:33 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 2 11:56 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 3 15:30 xprog + ./bin/tidb-server -V Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 + ./bin/tikv-server -V TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:19 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:19 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:19:48" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/cdc ./cmd/cdc go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/xdg/scram v1.0.5 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/time v0.7.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading golang.org/x/sys v0.28.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/goccy/go-json v0.10.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/go-mysql-org/go-mysql v1.12.0 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/swaggo/swag v1.16.3 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/golang/mock v1.6.0 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading google.golang.org/protobuf v1.35.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading filippo.io/edwards25519 v1.1.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.21.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/joomcode/errorx v1.0.1 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading golang.org/x/text v0.21.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/go-openapi/jsonreference v0.21.0 go: downloading github.com/go-openapi/swag v0.23.0 go: downloading github.com/go-openapi/jsonpointer v0.21.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/apache/thrift v0.16.0 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading k8s.io/apimachinery v0.29.11 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_kafka_consumer ']' + make kafka_consumer CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:22:53" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer go: downloading github.com/confluentinc/confluent-kafka-go/v2 v2.3.0 go: downloading github.com/DataDog/zstd v1.5.5 + '[' -f ./bin/cdc_storage_consumer ']' + make storage_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:35" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:43" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:43" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:43" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:43" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/oauth2-server ./cmd/oauth2-server/main.go go: downloading github.com/go-oauth2/oauth2/v4 v4.5.2 go: downloading github.com/tidwall/buntdb v1.3.0 go: downloading github.com/tidwall/rtred v0.1.2 go: downloading github.com/tidwall/gjson v1.14.3 go: downloading github.com/tidwall/match v1.1.1 go: downloading github.com/tidwall/grect v0.1.4 go: downloading github.com/tidwall/tinyqueue v0.1.1 go: downloading github.com/tidwall/pretty v1.2.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:43" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/sync_diff_inspector ./sync_diff_inspector go: downloading github.com/olekukonko/tablewriter v0.0.5 $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20241224111350-ad2e4f976c7c go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading github.com/glebarez/sqlite v1.7.0 go: downloading go.uber.org/dig v1.13.0 go: downloading gorm.io/driver/mysql v1.5.7 go: downloading github.com/glebarez/go-sqlite v1.21.2 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading modernc.org/sqlite v1.27.0 go: downloading modernc.org/libc v1.37.1 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading modernc.org/memory v1.7.2 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:43" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v9.0.0-beta.1-10-ga306f4894" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2025-04-04 07:25:43" -X "github.com/pingcap/tiflow/pkg/version.GitHash=a306f48949e037965a7c1963f500f7e0abbfba51" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v9.0.0-beta.1-10-ga306f4894"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.4G drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:31 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:19 .. -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:31 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:31 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:26 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:26 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:25 cdc_storage_consumer -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:26 oauth2-server -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:26 sync_diff_inspector + ./bin/cdc version Release Version: v9.0.0-beta.1-10-ga306f4894 Git Commit Hash: a306f48949e037965a7c1963f500f7e0abbfba51 Git Branch: HEAD UTC Build Time: 2025-04-04 07:25:43 Go Version: go1.23.4 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-test/rev-4c631d5-a306f48) 1485649920 bytes in 19.74 secs (75266185 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-region-bench ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/pd-ut ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ../third_party_download/bin/xprog ./bin/ + ls -alh ./bin total 3.3G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 4 15:32 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 4 15:19 .. drwxr-sr-x 2 jenkins jenkins 4.0K Apr 4 15:32 bin -rwxr-xr-x 1 jenkins jenkins 231M Apr 4 15:31 cdc -rwxr-xr-x 1 jenkins jenkins 364M Apr 4 15:31 cdc.test -rwxr-xr-x 1 jenkins jenkins 269M Apr 4 15:26 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 200M Apr 4 15:26 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 199M Apr 4 15:25 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:32 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 4 15:32 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:32 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 4 15:32 jq drwxr-sr-x 3 jenkins jenkins 4.0K Apr 4 15:32 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:32 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1.2M Apr 4 15:32 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 4 15:32 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 353K Apr 4 15:32 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 4 15:32 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 4 15:32 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.2M Apr 4 15:32 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 346M Apr 4 15:32 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Apr 4 15:32 minio -rwxr-xr-x 1 jenkins jenkins 13M Apr 4 15:26 oauth2-server -rwxr-xr-x 1 jenkins jenkins 41M Apr 4 15:32 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 44M Apr 4 15:32 pd-ctl -rwxr-xr-x 1 jenkins jenkins 40M Apr 4 15:32 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:32 pd-recover -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:32 pd-region-bench -rwxr-xr-x 1 jenkins jenkins 115M Apr 4 15:32 pd-server -rwxr-xr-x 1 jenkins jenkins 37M Apr 4 15:32 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 15M Apr 4 15:32 pd-ut -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:32 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K Apr 4 15:32 share -rwxr-xr-x 1 jenkins jenkins 35M Apr 4 15:32 stores-dump -rwxr-xr-x 1 jenkins jenkins 145M Apr 4 15:26 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 241M Apr 4 15:32 tidb-server -rwxr-xr-x 1 jenkins jenkins 443M Apr 4 15:32 tiflash -rwxr-xr-x 1 jenkins jenkins 373M Apr 4 15:32 tikv-server -rwxr-xr-x 1 jenkins jenkins 2.3M Apr 4 15:32 xprog [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 46.96 secs (85836752 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G18') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G19') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G20') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G21') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G18') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G19') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G20') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G21') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b Agent pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-kb04p-jr3kq --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "6ab80a5bcbfb6b42933ef0a10b51706e9e8312eb" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-kb04p" name: "pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6c621e3d; decorates RemoteLauncher[hudson.remoting.Channel@4e18f44a:JNLP4-connect connection from 10.233.76.86/10.233.76.86:50376] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate Agent pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-cdl5f-btd71 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "3e71d62a88d8758f8886b60814d2e983b4e90dd7" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-cdl5f" name: "pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" [Pipeline] { Running on pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] { [Pipeline] node [Pipeline] checkout The recommended git tool is: git Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3f8ce0ff; decorates RemoteLauncher[hudson.remoting.Channel@67e5ee13:JNLP4-connect connection from 10.233.72.215/10.233.72.215:39152] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] withEnv [Pipeline] { Agent pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-jln6z-32lr7 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "7daed5a29ba0870562533562318a576475e08b30" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-jln6z" name: "pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git Running on pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] stage [Pipeline] { (Test) The recommended git tool is: git [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] cache Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-ftv0p-w18v8 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f4a2a5de6e7349c2e004039adc4c884968ce0c5a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-ftv0p" name: "pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@64687a7e; decorates RemoteLauncher[hudson.remoting.Channel@791842e2:JNLP4-connect connection from 10.233.78.76/10.233.78.76:54048] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-djxkt-cmhg4 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "de16eb1307d94f7af7cc8bfb152de9868e98c3c2" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-djxkt" name: "pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Agent pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-v1kts-zfc7f --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e9e54d9af72e2a7f6133ae80aa10ca791f0434c8" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-v1kts" name: "pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-b0k0m-r6x1k --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "f069853562cccfd456d1004f09e4407721c7d498" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-b0k0m" name: "pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-96b93-8j6sc --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "1e0125e364920e9aabe5ed841b057aa0c03cb45a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-96b93" name: "pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-q97cb-x6ms7 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "91ec402fe4b6fa7910561f1dac5f8bc301718916" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-q97cb" name: "pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.89 secs (511117780 bytes/sec) [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] sh [Pipeline] { [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G00 [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node Obtained pipelines/pingcap/tiflow/latest/pod-pull_cdc_integration_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] { Run cases: down_db_require_secure_transport bdr_mode capture_suicide_while_balance_table syncpoint syncpoint_check_ts server_config_compatibility changefeed_dup_error_restart safe_mode kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro debezium lossy_ddl storage_csv_update csv_storage_update_pk_clustered csv_storage_update_pk_nonclustered GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-kb04p HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=905ca971-9fe1-4245-90d7-2d072adab497 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G00 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-kb04p pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/down_db_require_secure_transport/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] container The recommended git tool is: git [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] { The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4 The recommended git tool is: git Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1 [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6 [Pipeline] checkout Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6 The recommended git tool is: git [Pipeline] podTemplate The recommended git tool is: git [Pipeline] { [Pipeline] node [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] podTemplate [Pipeline] { [Pipeline] node [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5a289b0f; decorates RemoteLauncher[hudson.remoting.Channel@c112b72:JNLP4-connect connection from 10.233.76.230/10.233.76.230:49916] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@58e28e39; decorates RemoteLauncher[hudson.remoting.Channel@4cb6dc49:JNLP4-connect connection from 10.233.78.239/10.233.78.239:58790] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6e7b1830; decorates RemoteLauncher[hudson.remoting.Channel@4bbbd72b:JNLP4-connect connection from 10.233.76.160/10.233.76.160:39194] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@182e978b; decorates RemoteLauncher[hudson.remoting.Channel@46e6d04:JNLP4-connect connection from 10.233.72.134/10.233.72.134:54024] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6d58c3b2; decorates RemoteLauncher[hudson.remoting.Channel@6d336e53:JNLP4-connect connection from 10.233.66.133/10.233.66.133:54664] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@a74ab78; decorates RemoteLauncher[hudson.remoting.Channel@397b6c6c:JNLP4-connect connection from 10.233.72.18/10.233.72.18:55898] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Created Pod: kubernetes jenkins-tiflow/pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Agent pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-smdjj-qrvfs --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ef835d9db2c861969380cc851c7d849887cce7fd" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-smdjj" name: "pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Running on pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-qfqk9-cn0b1 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "649c95a6ab404e7303e49d361ba7f8fb6704d34a" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-qfqk9" name: "pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Running on pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Agent pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-8bgqv-08rx2 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "21def1f96dfe9018ad42e97b4920789b3cb2695f" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-8bgqv" name: "pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-zpm8c-13z7t --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "1c5895fe21a325673d291acd087f9272a448d376" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-zpm8c" name: "pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Agent pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-txjnm-sgfcp --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "be50f5396929aafb60fc567157b80eb7ac057d65" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-txjnm" name: "pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-n8g2x-37nt4 Agent pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-qth0n-2cmqp --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "08ee886486951837520de1bdeec0cec0d03184f3" --- apiVersion: "v1" kind: "Pod" metadata: annotations: jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-n8g2x" buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: name: "pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ef6445afccb1f27c0c69d61e39412db2bebca85f" nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-qth0n" name: "pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4" operator: "In" values: - "amd64" containers: namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: name: "report" resources: limits: memory: "256Mi" cpu: "100m" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" value: "pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl" - name: "JENKINS_NAME" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" value: "pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4" - name: "JENKINS_NAME" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" value: "pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 - emptyDir: medium: "" name: "workspace-volume" volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Running on pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-rkmnv-dl2gj --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "13525fbb198fe3bd4723b33bab160d98a205beb7" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-rkmnv" name: "pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-0vf91-8tdb4 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "fa1f801b55254a3ae65d3dd950251e80f6802031" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-0vf91" name: "pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Running on pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Agent pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-vsll5-3qk5x --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "6a78c0973057b72ac81fcf8dadedd1a49f5e45c5" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-vsll5" name: "pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-m04ht-46fh4 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "6b116d4c04a11e27a496a6bf4c9f39cf9b86ba41" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-m04ht" name: "pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-5fgk4-h679d --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "34761c1b0711255d43989b996dec9878c439f295" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-5fgk4" name: "pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test Agent pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6 is provisioned from template pingcap_tiflow_pull_cdc_integration_test_3827-zxhg8-9pc98 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" runUrl: "job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "51f1c083896c6922497951085771ac2aefb83001" jenkins/label: "pingcap_tiflow_pull_cdc_integration_test_3827-zxhg8" name: "pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/rocky8_golang-1.23:latest" name: "golang" resources: limits: memory: "32Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6" - name: "JENKINS_NAME" value: "pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6 in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test start tidb cluster in /tmp/tidb_cdc_test/down_db_require_secure_transport Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) config file /tmp/tidb_cdc_test/down_db_require_secure_transport/tidb-config-1743752006545897494.toml contained invalid configuration options: security.socket; check TiDB manual to make sure this option has not been deprecated and removed from your TiDB version if the option does not appear to be a typo Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 8.55 secs (471663606 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357818566981530 TiDB Cluster ID. tikv_gc_leader_uuid 657fee057340007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:1208, start at 2025-04-04 15:33:29.174019332 +0800 CST m=+2.705349663 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:29.181 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:29.165 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:29.165 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357818566981530 TiDB Cluster ID. tikv_gc_leader_uuid 657fee057340007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:1208, start at 2025-04-04 15:33:29.174019332 +0800 CST m=+2.705349663 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:29.181 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:29.165 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:29.165 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357826309559552 TiDB Cluster ID. tikv_gc_leader_uuid 657fee058ac000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:1276, start at 2025-04-04 15:33:29.288030545 +0800 CST m=+2.728804968 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:29.295 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:29.259 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:29.259 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/down_db_require_secure_transport/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:33:32 CST 2025] <<<<<< START cdc server in down_db_require_secure_transport case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.down_db_require_secure_transport.26612663.out server --log-file /tmp/tidb_cdc_test/down_db_require_secure_transport/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/down_db_require_secure_transport/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 6.61 secs (609730948 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:33:36 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/476360be-68f5-432b-ab5e-21c1a1ce57ba {"id":"476360be-68f5-432b-ab5e-21c1a1ce57ba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752013} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb7f332da 476360be-68f5-432b-ab5e-21c1a1ce57ba /tidb/cdc/default/default/upstream/7489357818566981530 {"id":7489357818566981530,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/476360be-68f5-432b-ab5e-21c1a1ce57ba {"id":"476360be-68f5-432b-ab5e-21c1a1ce57ba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752013} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb7f332da 476360be-68f5-432b-ab5e-21c1a1ce57ba /tidb/cdc/default/default/upstream/7489357818566981530 {"id":7489357818566981530,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/476360be-68f5-432b-ab5e-21c1a1ce57ba {"id":"476360be-68f5-432b-ab5e-21c1a1ce57ba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752013} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb7f332da 476360be-68f5-432b-ab5e-21c1a1ce57ba /tidb/cdc/default/default/upstream/7489357818566981530 {"id":7489357818566981530,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G01 Run cases: http_api http_api_tls api_v2 http_api_tls_with_user_auth cli_tls_with_auth debezium_basic kafka_simple_basic kafka_simple_basic_avro kafka_simple_handle_key_only kafka_simple_handle_key_only_avro kafka_simple_claim_check kafka_simple_claim_check_avro canal_json_adapter_compatibility canal_json_basic canal_json_content_compatible multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-cdl5f HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=28fbcf25-da72-4bc5-8e18-d3211a4af238 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G01 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8 pingcap_tiflow_pull_cdc_integration_test_3827-cdl5f PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G04 [Pipeline] { [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Collecting requests==2.26.0 [Pipeline] { Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-jln6z HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=22b8f221-d716-458d-91d9-fc9e76f9d3a7 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G04 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-jln6z pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { Using cached https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Collecting certifi>=2017.4.17 (from requests==2.26.0) [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.down_db_require_secure_transport.cli.2730.out cli changefeed create '--sink-uri=mysql://root@127.0.0.1:3306/?ssl-ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem&ssl-key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem&ssl-cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem' --changefeed-id=down-db-require-secure-transport [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) Downloading https://files.pythonhosted.org/packages/38/fc/bce832fd4fd99766c04d1ee0eead6b0ec6486fb100ae5e74c1d91292b982/certifi-2025.1.31-py3-none-any.whl (166kB) Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) [Pipeline] stage [Pipeline] { (Test) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2f335422; decorates RemoteLauncher[hudson.remoting.Channel@7237f609:JNLP4-connect connection from 10.233.77.41/10.233.77.41:44330] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6f4415a5; decorates RemoteLauncher[hudson.remoting.Channel@2fabf823:JNLP4-connect connection from 10.233.78.216/10.233.78.216:41444] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4a05a914; decorates RemoteLauncher[hudson.remoting.Channel@6e8875a3:JNLP4-connect connection from 10.233.84.119/10.233.84.119:42208] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@797bda5c; decorates RemoteLauncher[hudson.remoting.Channel@40aeea2e:JNLP4-connect connection from 10.233.86.113/10.233.86.113:50692] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] timeout Timeout set to expire in 40 min Create changefeed successfully! ID: down-db-require-secure-transport Info: {"upstream_id":7489357818566981530,"namespace":"default","id":"down-db-require-secure-transport","sink_uri":"mysql://root@127.0.0.1:3306/?ssl-ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem\u0026ssl-key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem\u0026ssl-cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem","create_time":"2025-04-04T15:33:38.521771268+08:00","start_ts":457114129102274567,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114129102274567,"checkpoint_ts":457114129102274567,"checkpoint_time":"2025-04-04 15:33:38.365"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4d03689d; decorates RemoteLauncher[hudson.remoting.Channel@6e802507:JNLP4-connect connection from 10.233.107.186/10.233.107.186:34334] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7d5fea51; decorates RemoteLauncher[hudson.remoting.Channel@31636348:JNLP4-connect connection from 10.233.71.81/10.233.71.81:36334] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] withCredentials Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@67aa3e47; decorates RemoteLauncher[hudson.remoting.Channel@6721a8f4:JNLP4-connect connection from 10.233.69.127/10.233.69.127:53970] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6c8eb227; decorates RemoteLauncher[hudson.remoting.Channel@16eaae8f:JNLP4-connect connection from 10.233.108.67/10.233.108.67:35426] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@148e309; decorates RemoteLauncher[hudson.remoting.Channel@47a2ff23:JNLP4-connect connection from 10.233.68.36/10.233.68.36:36106] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Installing collected packages: certifi, requests Found existing installation: certifi 2024.8.30 Uninstalling certifi-2024.8.30: Successfully uninstalled certifi-2024.8.30 Found existing installation: requests 2.27.1 Uninstalling requests-2.27.1: Successfully uninstalled requests-2.27.1 Successfully installed certifi-2025.1.31 requests-2.26.0 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@290c0863; decorates RemoteLauncher[hudson.remoting.Channel@4e2a3232:JNLP4-connect connection from 10.233.127.132/10.233.127.132:38676] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@26cc22b4; decorates RemoteLauncher[hudson.remoting.Channel@52a565fc:JNLP4-connect connection from 10.233.105.5/10.233.105.5:33596] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { [Pipeline] { [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@29aceff6; decorates RemoteLauncher[hudson.remoting.Channel@40126725:JNLP4-connect connection from 10.233.106.123/10.233.106.123:57578] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] { [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@25fb5765; decorates RemoteLauncher[hudson.remoting.Channel@5b4d4477:JNLP4-connect connection from 10.233.76.29/10.233.76.29:55354] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 The 1 times to try to start tidb cluster... [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] cache Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 + set +x Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Avoid second fetch > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" Avoid second fetch Checking out Revision b2a59508c11c86ba701c4af25fc536a9475b82c6 (origin/main) Commit message: "feat(prow-jobs/pingcap-qe/ci): add periodic job to crawl pull requests and issues (#3449)" start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 table down_db_require_secure_transport.finishmark not exists for 1-th check, retry later > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f b2a59508c11c86ba701c4af25fc536a9475b82c6 # timeout=10 Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/http_api Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table down_db_require_secure_transport.finishmark exists [Fri Apr 4 15:33:44 CST 2025] <<<<<< run test case down_db_require_secure_transport success! >>>>>> Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357904741105224 TiDB Cluster ID. tikv_gc_leader_uuid 657fee1873c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:1225, start at 2025-04-04 15:33:48.629881971 +0800 CST m=+2.023330700 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:48.639 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:48.623 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:48.623 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.62 secs (529104583 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357904741105224 TiDB Cluster ID. tikv_gc_leader_uuid 657fee1873c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:1225, start at 2025-04-04 15:33:48.629881971 +0800 CST m=+2.023330700 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:48.639 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:48.623 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:48.623 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357912908722125 TiDB Cluster ID. tikv_gc_leader_uuid 657fee188cc0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:1296, start at 2025-04-04 15:33:48.735788851 +0800 CST m=+2.057971730 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:48.742 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:48.723 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:48.723 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357915444116930 TiDB Cluster ID. tikv_gc_leader_uuid 657fee19da40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:1443, start at 2025-04-04 15:33:50.067069216 +0800 CST m=+2.707012275 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:50.075 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:50.057 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:50.057 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357915444116930 TiDB Cluster ID. tikv_gc_leader_uuid 657fee19da40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:1443, start at 2025-04-04 15:33:50.067069216 +0800 CST m=+2.707012275 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:50.075 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:50.057 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:50.057 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357916348074780 TiDB Cluster ID. tikv_gc_leader_uuid 657fee19c60000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:1509, start at 2025-04-04 15:33:50.014087963 +0800 CST m=+2.589783163 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:35:50.063 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:33:50.026 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:23:50.026 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_api/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2677.out cli tso query --pd=http://127.0.0.1:2379 [Fri Apr 4 15:33:53 CST 2025] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.27512753.out server --log-file /tmp/tidb_cdc_test/http_api/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114133033123841 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114133033123841 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:33:54 CST 2025] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.27192721.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:33:56 GMT < Content-Length: 854 < Content-Type: text/plain; charset=utf-8 < { [854 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 {"id":"758c8d38-baa9-47fd-a2ed-c1ad451cd6b4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752034} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a983c 758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 {"id":"758c8d38-baa9-47fd-a2ed-c1ad451cd6b4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752034} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a983c 758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 {"id":"758c8d38-baa9-47fd-a2ed-c1ad451cd6b4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752034} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a983c 758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:33:57 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98fca824-6c20-4177-9b2a-6c28e50d5513 {"id":"98fca824-6c20-4177-9b2a-6c28e50d5513","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752035} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb845cde9 98fca824-6c20-4177-9b2a-6c28e50d5513 /tidb/cdc/default/default/upstream/7489357904741105224 {"id":7489357904741105224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98fca824-6c20-4177-9b2a-6c28e50d5513 {"id":"98fca824-6c20-4177-9b2a-6c28e50d5513","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752035} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb845cde9 98fca824-6c20-4177-9b2a-6c28e50d5513 /tidb/cdc/default/default/upstream/7489357904741105224 {"id":7489357904741105224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98fca824-6c20-4177-9b2a-6c28e50d5513 {"id":"98fca824-6c20-4177-9b2a-6c28e50d5513","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752035} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb845cde9 98fca824-6c20-4177-9b2a-6c28e50d5513 /tidb/cdc/default/default/upstream/7489357904741105224 {"id":7489357904741105224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2779.out cli changefeed create --start-ts=457114133033123841 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/foreign_key/conf/cf.toml Create changefeed successfully! ID: cd1ea019-be8f-4bf0-bc3f-e9134c6c8bda Info: {"upstream_id":7489357904741105224,"namespace":"default","id":"cd1ea019-be8f-4bf0-bc3f-e9134c6c8bda","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:33:58.294578483+08:00","start_ts":457114133033123841,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!foreign_key.t12"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114133033123841,"checkpoint_ts":457114133033123841,"checkpoint_time":"2025-04-04 15:33:53.360"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... "is-owner": true, run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x owner pid: 2756 owner id 758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 default [Fri Apr 4 15:33:59 CST 2025] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.28732875.out server --log-file /tmp/tidb_cdc_test/http_api/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table foreign_key.finish_mark not exists for 1-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 9.03 secs (446236282 bytes/sec) [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/bdr_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table foreign_key.finish_mark not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:34:02 GMT < Content-Length: 1305 < Content-Type: text/plain; charset=utf-8 < { [1305 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 {"id":"758c8d38-baa9-47fd-a2ed-c1ad451cd6b4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752034} /tidb/cdc/default/__cdc_meta__/capture/ce30258d-3b27-4cfa-9030-d4b54b991bf6 {"id":"ce30258d-3b27-4cfa-9030-d4b54b991bf6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752040} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a983c 758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a987c ce30258d-3b27-4cfa-9030-d4b54b991bf6 /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 {"id":"758c8d38-baa9-47fd-a2ed-c1ad451cd6b4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752034} /tidb/cdc/default/__cdc_meta__/capture/ce30258d-3b27-4cfa-9030-d4b54b991bf6 {"id":"ce30258d-3b27-4cfa-9030-d4b54b991bf6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752040} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a983c 758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a987c ce30258d-3b27-4cfa-9030-d4b54b991bf6 /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 {"id":"758c8d38-baa9-47fd-a2ed-c1ad451cd6b4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752034} /tidb/cdc/default/__cdc_meta__/capture/ce30258d-3b27-4cfa-9030-d4b54b991bf6 {"id":"ce30258d-3b27-4cfa-9030-d4b54b991bf6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752040} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a983c 758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a987c ce30258d-3b27-4cfa-9030-d4b54b991bf6 /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "758c8d38-baa9-47fd-a2ed-c1ad451cd6b4 default" | grep id Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release "id": "ce30258d-3b27-4cfa-9030-d4b54b991bf6", run task successfully table foreign_key.finish_mark not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) capture_id: ce30258d-3b27-4cfa-9030-d4b54b991bf6 pass test: check health pass test: get status table foreign_key.finish_mark not exists for 4-th check, retry later pass test: create changefeed ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 5-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.26 secs (554966696 bytes/sec) [Pipeline] { [Pipeline] cache table foreign_key.finish_mark not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357996347412249 TiDB Cluster ID. tikv_gc_leader_uuid 657fee2b6680011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:3811, start at 2025-04-04 15:34:08.05644992 +0800 CST m=+2.155870672 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:08.064 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:08.026 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:08.026 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357996347412249 TiDB Cluster ID. tikv_gc_leader_uuid 657fee2b6680011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:3811, start at 2025-04-04 15:34:08.05644992 +0800 CST m=+2.155870672 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:08.064 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:08.026 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:08.026 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489357994034685260 TiDB Cluster ID. tikv_gc_leader_uuid 657fee2b41c0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:3884, start at 2025-04-04 15:34:07.882138645 +0800 CST m=+1.913298197 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:07.888 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:07.929 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:07.929 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.simple1 not exists for 1-th check, retry later [Fri Apr 4 15:34:12 CST 2025] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.51175119.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.simple1 exists table foreign_key.finish_mark not exists for 7-th check, retry later pass test: list changefeed pass test: get changefeed table foreign_key.finish_mark not exists for 8-th check, retry later pass test: pause changefeed pass test: update changefeed + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:34:15 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d25c83a-0164-4486-9bdb-8ae2e2e44493 {"id":"5d25c83a-0164-4486-9bdb-8ae2e2e44493","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752052} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb89334d4 5d25c83a-0164-4486-9bdb-8ae2e2e44493 /tidb/cdc/default/default/upstream/7489357996347412249 {"id":7489357996347412249,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d25c83a-0164-4486-9bdb-8ae2e2e44493 {"id":"5d25c83a-0164-4486-9bdb-8ae2e2e44493","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752052} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb89334d4 5d25c83a-0164-4486-9bdb-8ae2e2e44493 /tidb/cdc/default/default/upstream/7489357996347412249 {"id":7489357996347412249,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5d25c83a-0164-4486-9bdb-8ae2e2e44493 {"id":"5d25c83a-0164-4486-9bdb-8ae2e2e44493","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752052} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb89334d4 5d25c83a-0164-4486-9bdb-8ae2e2e44493 /tidb/cdc/default/default/upstream/7489357996347412249 {"id":7489357996347412249,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:34:15 CST 2025] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.51725174.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdcdown.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_datadown --cluster-id default --addr 127.0.0.1:8400 --pd http://127.0.0.1:2479 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8400 failed: Connection refused * Failed to connect to 127.0.0.1 port 8400: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process 2756 exit for 1-th time... table foreign_key.finish_mark not exists for 9-th check, retry later wait process 2756 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2756) - No such process wait process 2756 exit for 3-th time... process 2756 already exit cdc.test cli capture list --server http://127.0.0.1:8301 2>&1 |grep ce30258d-3b27-4cfa-9030-d4b54b991bf6 -A1 | grep '"is-owner": true' table foreign_key.finish_mark not exists for 10-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 6.78 secs (594090949 bytes/sec) [Pipeline] { "is-owner": true, run task successfully [Fri Apr 4 15:34:18 CST 2025] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.31273129.out server --log-file /tmp/tidb_cdc_test/http_api/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] cache + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8400 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8400 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:34:18 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/924de7c4-62a1-452a-832c-87a13363a798 {"id":"924de7c4-62a1-452a-832c-87a13363a798","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752055} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffb8936a02 924de7c4-62a1-452a-832c-87a13363a798 /tidb/cdc/default/default/upstream/7489357994034685260 {"id":7489357994034685260,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/924de7c4-62a1-452a-832c-87a13363a798 {"id":"924de7c4-62a1-452a-832c-87a13363a798","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752055} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffb8936a02 924de7c4-62a1-452a-832c-87a13363a798 /tidb/cdc/default/default/upstream/7489357994034685260 {"id":7489357994034685260,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/924de7c4-62a1-452a-832c-87a13363a798 {"id":"924de7c4-62a1-452a-832c-87a13363a798","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752055} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffb8936a02 924de7c4-62a1-452a-832c-87a13363a798 /tidb/cdc/default/default/upstream/7489357994034685260 {"id":7489357994034685260,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.5230.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306 -c test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bdr_mode/conf/up.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489357996347412249,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306","create_time":"2025-04-04T15:34:18.924777379+08:00","start_ts":457114139708882948,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":true,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114139708882948,"checkpoint_ts":457114139708882948,"checkpoint_time":"2025-04-04 15:34:18.826"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table foreign_key.finish_mark not exists for 11-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.5268.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:4000 -c test-2 --server http://127.0.0.1:8400 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bdr_mode/conf/down.toml Create changefeed successfully! ID: test-2 Info: {"upstream_id":7489357994034685260,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:4000","create_time":"2025-04-04T15:34:20.742780738+08:00","start_ts":457114140181528582,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":true,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114140181528582,"checkpoint_ts":457114140181528582,"checkpoint_time":"2025-04-04 15:34:20.629"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:34:21 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/changefeed-test3 {UpstreamID:7489357915444116930 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2025-04-04 15:34:06.470588201 +0800 CST StartTs:457114136461705221 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017ad9e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114136461705226} {CheckpointTs:457114136920457242 MinTableBarrierTs:457114140144828420 AdminJobType:noop} changefeedID: default/changefeed-test1 {UpstreamID:7489357915444116930 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:34:06.26581025 +0800 CST StartTs:457114136396169221 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017ac240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114136409538565} {CheckpointTs:457114136920457242 MinTableBarrierTs:457114140144828420 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114140394127364, checkpointTs: 457114136920457242, state: Prepared span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114136920457242, checkpointTs: 457114136920457242, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e66802f-e5ed-46da-88e6-4be86bfb3e0b {"id":"2e66802f-e5ed-46da-88e6-4be86bfb3e0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752058} /tidb/cdc/default/__cdc_meta__/capture/ce30258d-3b27-4cfa-9030-d4b54b991bf6 {"id":"ce30258d-3b27-4cfa-9030-d4b54b991bf6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752040} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a987c ce30258d-3b27-4cfa-9030-d4b54b991bf6 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a9931 2e66802f-e5ed-46da-88e6-4be86bfb3e0b /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:34:06.26581025+08:00","start-ts":457114136396169221,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114136409538565} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2025-04-04T15:34:06.385519026+08:00","start-ts":457114136435752966,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"stopped","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114138401832968} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2025-04-04T15:34:06.470588201+08:00","start-ts":457114136461705221,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114136461705226} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114140406972420,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114138309820417,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114140406972420,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e66802f-e5ed-46da-88e6-4be86bfb3e0b/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2e66802f-e5ed-46da-88e6-4be86bfb3e0b/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ce30258d-3b27-4cfa-9030-d4b54b991bf6/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ce30258d-3b27-4cfa-9030-d4b54b991bf6/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/changefeed-test3 {UpstreamID:7489357915444116930 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2025-04-04 15:34:06.470588201 +0800 CST StartTs:457114136461705221 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017ad9e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114136461705226} {CheckpointTs:457114136920457242 MinTableBarrierTs:457114140144828420 AdminJobType:noop} changefeedID: default/changefeed-test1 {UpstreamID:7489357915444116930 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:34:06.26581025 +0800 CST StartTs:457114136396169221 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017ac240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114136409538565} {CheckpointTs:457114136920457242 MinTableBarrierTs:457114140144828420 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114140394127364, checkpointTs: 457114136920457242, state: Prepared span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114136920457242, checkpointTs: 457114136920457242, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e66802f-e5ed-46da-88e6-4be86bfb3e0b {"id":"2e66802f-e5ed-46da-88e6-4be86bfb3e0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752058} /tidb/cdc/default/__cdc_meta__/capture/ce30258d-3b27-4cfa-9030-d4b54b991bf6 {"id":"ce30258d-3b27-4cfa-9030-d4b54b991bf6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752040} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a987c ce30258d-3b27-4cfa-9030-d4b54b991bf6 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a9931 2e66802f-e5ed-46da-88e6-4be86bfb3e0b /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:34:06.26581025+08:00","start-ts":457114136396169221,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114136409538565} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2025-04-04T15:34:06.385519026+08:00","start-ts":457114136435752966,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"stopped","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114138401832968} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2025-04-04T15:34:06.470588201+08:00","start-ts":457114136461705221,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114136461705226} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114140406972420,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114138309820417,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114140406972420,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e66802f-e5ed-46da-88e6-4be86bfb3e0b/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2e66802f-e5ed-46da-88e6-4be86bfb3e0b/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ce30258d-3b27-4cfa-9030-d4b54b991bf6/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ce30258d-3b27-4cfa-9030-d4b54b991bf6/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/changefeed-test3 {UpstreamID:7489357915444116930 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2025-04-04 15:34:06.470588201 +0800 CST StartTs:457114136461705221 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017ad9e0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114136461705226} {CheckpointTs:457114136920457242 MinTableBarrierTs:457114140144828420 AdminJobType:noop} changefeedID: default/changefeed-test1 {UpstreamID:7489357915444116930 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:34:06.26581025 +0800 CST StartTs:457114136396169221 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0017ac240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114136409538565} {CheckpointTs:457114136920457242 MinTableBarrierTs:457114140144828420 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114140394127364, checkpointTs: 457114136920457242, state: Prepared span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114136920457242, checkpointTs: 457114136920457242, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e66802f-e5ed-46da-88e6-4be86bfb3e0b {"id":"2e66802f-e5ed-46da-88e6-4be86bfb3e0b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752058} /tidb/cdc/default/__cdc_meta__/capture/ce30258d-3b27-4cfa-9030-d4b54b991bf6 {"id":"ce30258d-3b27-4cfa-9030-d4b54b991bf6","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752040} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a987c ce30258d-3b27-4cfa-9030-d4b54b991bf6 /tidb/cdc/default/__cdc_meta__/owner/223195ffb84a9931 2e66802f-e5ed-46da-88e6-4be86bfb3e0b /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:34:06.26581025+08:00","start-ts":457114136396169221,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114136409538565} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2025-04-04T15:34:06.385519026+08:00","start-ts":457114136435752966,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"stopped","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114138401832968} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7489357915444116930,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2025-04-04T15:34:06.470588201+08:00","start-ts":457114136461705221,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114136461705226} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114140406972420,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114138309820417,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":457114136920457242,"min-table-barrier-ts":457114140406972420,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e66802f-e5ed-46da-88e6-4be86bfb3e0b/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/2e66802f-e5ed-46da-88e6-4be86bfb3e0b/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ce30258d-3b27-4cfa-9030-d4b54b991bf6/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ce30258d-3b27-4cfa-9030-d4b54b991bf6/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489357915444116930 {"id":7489357915444116930,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"address": "127.0.0.1:8300"' + set +x table foreign_key.finish_mark not exists for 12-th check, retry later "address": "127.0.0.1:8300", run task successfully pass test: list changefeed pass test: get changefeed pass test: pause changefeed table foreign_key.finish_mark not exists for 13-th check, retry later pass test: resume changefeed pass test: rebalance table pass test: move table table foreign_key.finish_mark exists check diff successfully table foreign_key.t12 does not exists wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:34:28 CST 2025] <<<<<< run test case foreign_key success! >>>>>> pass test: get processors pass test: list processors pass test: set log level pass test: remove changefeed pass test: resign owner Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 8.72 secs (462496248 bytes/sec) [Pipeline] { [Pipeline] cache =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 9.72 secs (414594109 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G03 [Pipeline] sh start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Run cases: row_format drop_many_tables processor_stop_delay partition_table ddl_with_exists GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-ftv0p HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=2bfb6332-1af3-4cea-89ba-f0818edb4b24 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G03 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b pingcap_tiflow_pull_cdc_integration_test_3827-ftv0p PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G06 [Pipeline] sh Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-djxkt HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=ac9c5f0b-46a0-4365-870d-8be2c9be96b1 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G06 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-djxkt pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G07 [Pipeline] sh Run cases: kv_client_stream_reconnect cdc split_region GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-v1kts HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=345a784c-1161-4ab5-a02a-5d1c24371bda GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G07 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-v1kts pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G08 [Pipeline] sh Run cases: processor_err_chan changefeed_reconstruct multi_capture synced_status_with_redo GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-b0k0m HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=ec456f99-3eac-4b7c-bb94-2280dd432311 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G08 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-b0k0m pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G05 Run cases: charset_gbk ddl_manager multi_source vector GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-96b93 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=7133eb8b-9f1b-4179-9b3a-d32773036e9a GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G05 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9 pingcap_tiflow_pull_cdc_integration_test_3827-96b93 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G02 The 1 times to try to start tidb cluster... Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-q97cb HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=ac9ac7e4-7c7d-4c14-983e-fcad3810c815 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G02 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-q97cb pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] stage [Pipeline] { (Test) start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] timeout Timeout set to expire in 40 min Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_ddl Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] { Verifying downstream PD is started... [Pipeline] cache Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358177057954977 TiDB Cluster ID. tikv_gc_leader_uuid 657fee55c7c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:3994, start at 2025-04-04 15:34:51.442638369 +0800 CST m=+2.221131777 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:51.453 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:51.423 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:51.423 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358177057954977 TiDB Cluster ID. tikv_gc_leader_uuid 657fee55c7c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:3994, start at 2025-04-04 15:34:51.442638369 +0800 CST m=+2.221131777 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:51.453 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:51.423 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:51.423 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358182106806684 TiDB Cluster ID. tikv_gc_leader_uuid 657fee55eb40009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:4073, start at 2025-04-04 15:34:51.579108434 +0800 CST m=+2.289527134 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:51.586 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:51.565 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:51.565 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5306.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358194992327660 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5902c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:1218, start at 2025-04-04 15:34:54.776988864 +0800 CST m=+2.371548084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:54.786 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:54.781 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:54.781 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358194992327660 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5902c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:1218, start at 2025-04-04 15:34:54.776988864 +0800 CST m=+2.371548084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:54.786 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:54.781 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:54.781 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358192592287766 TiDB Cluster ID. tikv_gc_leader_uuid 657fee593440007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:1277, start at 2025-04-04 15:34:54.941697051 +0800 CST m=+2.464291830 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:54.951 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:54.929 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:54.929 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358197538927296 TiDB Cluster ID. tikv_gc_leader_uuid 657fee596640006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:1356, start at 2025-04-04 15:34:55.136737947 +0800 CST m=+2.184474783 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:55.145 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:55.129 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:55.129 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358197538927296 TiDB Cluster ID. tikv_gc_leader_uuid 657fee596640006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:1356, start at 2025-04-04 15:34:55.136737947 +0800 CST m=+2.184474783 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:55.145 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:55.129 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:55.129 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358195388076308 TiDB Cluster ID. tikv_gc_leader_uuid 657fee595900004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:1430, start at 2025-04-04 15:34:55.081121439 +0800 CST m=+2.039703432 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:55.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:55.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:55.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='457114149446746113 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114149446746113 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:34:57 CST 2025] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.53535355.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358196490336729 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5a918000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:1363, start at 2025-04-04 15:34:56.370570456 +0800 CST m=+2.426580361 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:56.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:56.376 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:56.376 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358196490336729 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5a918000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:1363, start at 2025-04-04 15:34:56.370570456 +0800 CST m=+2.426580361 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:56.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:56.376 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:56.376 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358201664899834 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5a9d80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:1426, start at 2025-04-04 15:34:56.387209468 +0800 CST m=+2.336410635 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:56.428 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:56.426 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:56.426 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358203854966258 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5c4300016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:1202, start at 2025-04-04 15:34:58.108650168 +0800 CST m=+3.583233715 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:58.117 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:58.110 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:58.110 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358203854966258 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5c4300016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:1202, start at 2025-04-04 15:34:58.108650168 +0800 CST m=+3.583233715 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:58.117 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:58.110 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:58.110 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358199068301437 TiDB Cluster ID. tikv_gc_leader_uuid 657fee5c2940002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:1271, start at 2025-04-04 15:34:57.96271395 +0800 CST m=+3.364003967 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:36:57.974 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:34:57.957 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:24:57.957 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2644.out cli tso query --pd=http://127.0.0.1:2379 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.28 secs (553866933 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:35:00 CST 2025] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.26622664.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='457114150274596865 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114150274596865 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:35:00 CST 2025] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.26852687.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:35:00 CST 2025] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/ProcessorAddTableError=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.27572759.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:00 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59 {"id":"105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752097} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb93679fb 105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59 /tidb/cdc/default/default/upstream/7489358177057954977 {"id":7489358177057954977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59 {"id":"105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752097} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb93679fb 105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59 /tidb/cdc/default/default/upstream/7489358177057954977 {"id":7489358177057954977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59 {"id":"105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752097} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb93679fb 105cc2e6-f05b-49a4-96c0-a1e5fb8ddb59 /tidb/cdc/default/default/upstream/7489358177057954977 {"id":7489358177057954977,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5415.out cli changefeed create --start-ts=457114149446746113 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table tidb_cdc.syncpoint_v1 exists table tidb_cdc.syncpoint_v1 exists table bdr_mode.finish_mark exists table bdr_mode.finish_mark not exists for 1-th check, retry later table bdr_mode.finish_mark not exists for 2-th check, retry later table bdr_mode.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... Create changefeed successfully! ID: 12e7cf33-3c6d-4211-9454-be4f10adc3c2 Info: {"upstream_id":7489358177057954977,"namespace":"default","id":"12e7cf33-3c6d-4211-9454-be4f10adc3c2","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:35:01.014009014+08:00","start_ts":457114149446746113,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114149446746113,"checkpoint_ts":457114149446746113,"checkpoint_time":"2025-04-04 15:34:55.973"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Fri Apr 4 15:35:00 CST 2025] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.26442646.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... pass test: get tso wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... wait process cdc.test exit for 4-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:35:02 CST 2025] <<<<<< run test case bdr_mode success! >>>>>> wait process cdc.test exit for 5-th time... + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 6-th time... cdc.test: no process found wait process cdc.test exit for 7-th time... process cdc.test already exit [Fri Apr 4 15:35:03 CST 2025] <<<<<< run test case http_api success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70d608fe-effe-43bd-addf-3fa229534f61 {"id":"70d608fe-effe-43bd-addf-3fa229534f61","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94accd9 70d608fe-effe-43bd-addf-3fa229534f61 /tidb/cdc/default/default/upstream/7489358196490336729 {"id":7489358196490336729,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70d608fe-effe-43bd-addf-3fa229534f61 {"id":"70d608fe-effe-43bd-addf-3fa229534f61","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94accd9 70d608fe-effe-43bd-addf-3fa229534f61 /tidb/cdc/default/default/upstream/7489358196490336729 {"id":7489358196490336729,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70d608fe-effe-43bd-addf-3fa229534f61 {"id":"70d608fe-effe-43bd-addf-3fa229534f61","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94accd9 70d608fe-effe-43bd-addf-3fa229534f61 /tidb/cdc/default/default/upstream/7489358196490336729 {"id":7489358196490336729,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6733f35c-4676-4eba-a464-f4daf45f46ac {"id":"6733f35c-4676-4eba-a464-f4daf45f46ac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb948dee1 6733f35c-4676-4eba-a464-f4daf45f46ac /tidb/cdc/default/default/upstream/7489358194992327660 {"id":7489358194992327660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6733f35c-4676-4eba-a464-f4daf45f46ac {"id":"6733f35c-4676-4eba-a464-f4daf45f46ac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb948dee1 6733f35c-4676-4eba-a464-f4daf45f46ac /tidb/cdc/default/default/upstream/7489358194992327660 {"id":7489358194992327660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6733f35c-4676-4eba-a464-f4daf45f46ac {"id":"6733f35c-4676-4eba-a464-f4daf45f46ac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb948dee1 6733f35c-4676-4eba-a464-f4daf45f46ac /tidb/cdc/default/default/upstream/7489358194992327660 {"id":7489358194992327660,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2745.out cli changefeed create --start-ts=457114150274596865 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d945c4ab-d646-4751-a24e-0ca05c57cf25 {"id":"d945c4ab-d646-4751-a24e-0ca05c57cf25","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b0330 d945c4ab-d646-4751-a24e-0ca05c57cf25 /tidb/cdc/default/default/upstream/7489358197538927296 {"id":7489358197538927296,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d945c4ab-d646-4751-a24e-0ca05c57cf25 {"id":"d945c4ab-d646-4751-a24e-0ca05c57cf25","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b0330 d945c4ab-d646-4751-a24e-0ca05c57cf25 /tidb/cdc/default/default/upstream/7489358197538927296 {"id":7489358197538927296,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d945c4ab-d646-4751-a24e-0ca05c57cf25 {"id":"d945c4ab-d646-4751-a24e-0ca05c57cf25","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752100} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b0330 d945c4ab-d646-4751-a24e-0ca05c57cf25 /tidb/cdc/default/default/upstream/7489358197538927296 {"id":7489358197538927296,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 463052e4-fdb9-4e34-b2d6-368ffb142ef8 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=463052e4-fdb9-4e34-b2d6-368ffb142ef8 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 463052e4-fdb9-4e34-b2d6-368ffb142ef8 -s + info='{ "upstream_id": 7489358197538927296, "namespace": "default", "id": "463052e4-fdb9-4e34-b2d6-368ffb142ef8", "state": "normal", "checkpoint_tso": 457114151414136833, "checkpoint_time": "2025-04-04 15:35:03.478", "error": null }' + echo '{ "upstream_id": 7489358197538927296, "namespace": "default", "id": "463052e4-fdb9-4e34-b2d6-368ffb142ef8", "state": "normal", "checkpoint_tso": 457114151414136833, "checkpoint_time": "2025-04-04 15:35:03.478", "error": null }' { "upstream_id": 7489358197538927296, "namespace": "default", "id": "463052e4-fdb9-4e34-b2d6-368ffb142ef8", "state": "normal", "checkpoint_tso": 457114151414136833, "checkpoint_time": "2025-04-04 15:35:03.478", "error": null } ++ echo '{' '"upstream_id":' 7489358197538927296, '"namespace":' '"default",' '"id":' '"463052e4-fdb9-4e34-b2d6-368ffb142ef8",' '"state":' '"normal",' '"checkpoint_tso":' 457114151414136833, '"checkpoint_time":' '"2025-04-04' '15:35:03.478",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358197538927296, '"namespace":' '"default",' '"id":' '"463052e4-fdb9-4e34-b2d6-368ffb142ef8",' '"state":' '"normal",' '"checkpoint_tso":' 457114151414136833, '"checkpoint_time":' '"2025-04-04' '15:35:03.478",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully Create changefeed successfully! ID: 5cba0386-4fbf-4f7e-abac-2eb20f60ca56 Info: {"upstream_id":7489358194992327660,"namespace":"default","id":"5cba0386-4fbf-4f7e-abac-2eb20f60ca56","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:35:04.126122934+08:00","start_ts":457114150274596865,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114150274596865,"checkpoint_ts":457114150274596865,"checkpoint_time":"2025-04-04 15:34:59.131"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:04 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5886f939-65a8-452e-be6b-becb3dda8c7f {"id":"5886f939-65a8-452e-be6b-becb3dda8c7f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752101} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b2bd6 5886f939-65a8-452e-be6b-becb3dda8c7f /tidb/cdc/default/default/upstream/7489358203854966258 {"id":7489358203854966258,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5886f939-65a8-452e-be6b-becb3dda8c7f {"id":"5886f939-65a8-452e-be6b-becb3dda8c7f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752101} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b2bd6 5886f939-65a8-452e-be6b-becb3dda8c7f /tidb/cdc/default/default/upstream/7489358203854966258 {"id":7489358203854966258,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5886f939-65a8-452e-be6b-becb3dda8c7f {"id":"5886f939-65a8-452e-be6b-becb3dda8c7f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752101} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b2bd6 5886f939-65a8-452e-be6b-becb3dda8c7f /tidb/cdc/default/default/upstream/7489358203854966258 {"id":7489358203854966258,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x ***************** properties ***************** "operationcount"="0" "updateproportion"="0" "mysql.user"="root" "mysql.port"="4000" "threadcount"="10" "mysql.host"="127.0.0.1" "recordcount"="50" "readproportion"="0" "insertproportion"="0" "dotransactions"="false" "readallfields"="true" "scanproportion"="0" "mysql.db"="consistent_replicate_ddl" "fieldcount"="3" "requestdistribution"="uniform" "workload"="core" ********************************************** Run finished, takes 12.651428ms INSERT - Takes(s): 0.0, Count: 49, OPS: 7024.7, Avg(us): 2284, Min(us): 1159, Max(us): 5946, 95th(us): 6000, 99th(us): 6000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table consistent_replicate_ddl.usertable1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 6.74 secs (597988352 bytes/sec) [Pipeline] { [Pipeline] cache check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358195516750468 TiDB Cluster ID. tikv_gc_leader_uuid 657fee6672c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:902, start at 2025-04-04 15:35:08.539016998 +0800 CST m=+13.983881795 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:08.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:08.541 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:08.541 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358195516750468 TiDB Cluster ID. tikv_gc_leader_uuid 657fee6672c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:902, start at 2025-04-04 15:35:08.539016998 +0800 CST m=+13.983881795 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:08.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:08.541 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:08.541 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358205430616474 TiDB Cluster ID. tikv_gc_leader_uuid 657fee667f80004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:977, start at 2025-04-04 15:35:08.547845786 +0800 CST m=+13.882279843 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:08.556 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:08.542 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:08.542 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... table consistent_replicate_ddl.usertable1 exists table consistent_replicate_ddl.usertable2 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358200379903311 TiDB Cluster ID. tikv_gc_leader_uuid 657fee666dc0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:893, start at 2025-04-04 15:35:08.481407604 +0800 CST m=+14.896231193 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:08.490 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:08.471 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:08.471 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358200379903311 TiDB Cluster ID. tikv_gc_leader_uuid 657fee666dc0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:893, start at 2025-04-04 15:35:08.481407604 +0800 CST m=+14.896231193 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:08.490 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:08.471 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:08.471 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358202783920809 TiDB Cluster ID. tikv_gc_leader_uuid 657fee66590000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:969, start at 2025-04-04 15:35:08.418978782 +0800 CST m=+14.652551490 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:08.428 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:08.437 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:08.437 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:35:10 CST 2025] <<<<<< run test case processor_err_chan success! >>>>>> [Fri Apr 4 15:35:11 CST 2025] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.26942696.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_ddl.usertable2 not exists for 2-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2665.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: mysql... <<================= WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Requirement already up-to-date: requests==2.26.0 in /usr/local/lib/python3.6/site-packages Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) table consistent_replicate_ddl.usertable2 exists table consistent_replicate_ddl.usertable3 not exists for 1-th check, retry later Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) The 1 times to try to start tidb cluster... + set +x + tso='457114153784967169 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114153784967169 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "insertproportion"="0" "mysql.db"="sink_retry" "readallfields"="true" "mysql.host"="127.0.0.1" "recordcount"="10" "threadcount"="2" "workload"="core" "dotransactions"="false" "readproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "mysql.port"="4000" "updateproportion"="0" "operationcount"="0" "scanproportion"="0" ********************************************** Run finished, takes 9.004914ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1835.5, Avg(us): 1688, Min(us): 1008, Max(us): 3443, 95th(us): 4000, 99th(us): 4000 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:14 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/782bbe5b-490d-478b-9350-3a6360ae2b50 {"id":"782bbe5b-490d-478b-9350-3a6360ae2b50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752111} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94a87fb 782bbe5b-490d-478b-9350-3a6360ae2b50 /tidb/cdc/default/default/upstream/7489358195516750468 {"id":7489358195516750468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/782bbe5b-490d-478b-9350-3a6360ae2b50 {"id":"782bbe5b-490d-478b-9350-3a6360ae2b50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752111} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94a87fb 782bbe5b-490d-478b-9350-3a6360ae2b50 /tidb/cdc/default/default/upstream/7489358195516750468 {"id":7489358195516750468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/782bbe5b-490d-478b-9350-3a6360ae2b50 {"id":"782bbe5b-490d-478b-9350-3a6360ae2b50","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752111} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94a87fb 782bbe5b-490d-478b-9350-3a6360ae2b50 /tidb/cdc/default/default/upstream/7489358195516750468 {"id":7489358195516750468,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Fri Apr 4 15:35:14 CST 2025] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.27252727.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 9ea97a95-e4c5-4c72-8c4d-80ff5d6d29a0 Info: {"upstream_id":7489358195516750468,"namespace":"default","id":"9ea97a95-e4c5-4c72-8c4d-80ff5d6d29a0","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:35:14.401306648+08:00","start_ts":457114153423208449,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114153423208449,"checkpoint_ts":457114153423208449,"checkpoint_time":"2025-04-04 15:35:11.142"} table row_format.finish_mark not exists for 1-th check, retry later table consistent_replicate_ddl.usertable3 not exists for 2-th check, retry later table row_format.finish_mark not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/capture_suicide_while_balance_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:17 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6b79a6ca-8d8a-4b0b-8253-12ff419c48cb {"id":"6b79a6ca-8d8a-4b0b-8253-12ff419c48cb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752114} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9474110 6b79a6ca-8d8a-4b0b-8253-12ff419c48cb /tidb/cdc/default/default/upstream/7489358200379903311 {"id":7489358200379903311,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6b79a6ca-8d8a-4b0b-8253-12ff419c48cb {"id":"6b79a6ca-8d8a-4b0b-8253-12ff419c48cb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752114} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9474110 6b79a6ca-8d8a-4b0b-8253-12ff419c48cb /tidb/cdc/default/default/upstream/7489358200379903311 {"id":7489358200379903311,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6b79a6ca-8d8a-4b0b-8253-12ff419c48cb {"id":"6b79a6ca-8d8a-4b0b-8253-12ff419c48cb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752114} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9474110 6b79a6ca-8d8a-4b0b-8253-12ff419c48cb /tidb/cdc/default/default/upstream/7489358200379903311 {"id":7489358200379903311,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2785.out cli changefeed create --start-ts=457114153784967169 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' Create changefeed successfully! ID: 2f1c7fad-82f4-412c-a050-d82b82add820 Info: {"upstream_id":7489358200379903311,"namespace":"default","id":"2f1c7fad-82f4-412c-a050-d82b82add820","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:35:17.615595648+08:00","start_ts":457114153784967169,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114153784967169,"checkpoint_ts":457114153784967169,"checkpoint_time":"2025-04-04 15:35:12.522"} PASS table consistent_replicate_ddl.usertable3 exists table consistent_replicate_ddl.usertable_bak not exists for 1-th check, retry later coverage: 2.4% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table row_format.finish_mark not exists for 3-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 8.87 secs (454173277 bytes/sec) [Pipeline] { + set +x [Pipeline] cache table consistent_replicate_ddl.usertable_bak not exists for 2-th check, retry later Verifying downstream PD is started... table row_format.finish_mark not exists for 4-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later table consistent_replicate_ddl.usertable_bak exists Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table charset_gbk_test1.t0 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table row_format.finish_mark not exists for 5-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later table row_format.finish_mark not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 2-th check, retry later table row_format.finish_mark not exists for 7-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358323780607030 TiDB Cluster ID. tikv_gc_leader_uuid 657fee76b180002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:6515, start at 2025-04-04 15:35:25.129366117 +0800 CST m=+2.211661307 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:25.138 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:25.137 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:25.137 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358323780607030 TiDB Cluster ID. tikv_gc_leader_uuid 657fee76b180002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:6515, start at 2025-04-04 15:35:25.129366117 +0800 CST m=+2.211661307 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:25.138 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:25.137 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:25.137 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358323039269270 TiDB Cluster ID. tikv_gc_leader_uuid 657fee76b40000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:6582, start at 2025-04-04 15:35:25.15903054 +0800 CST m=+2.164073736 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:25.167 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:25.136 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:25.136 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... table row_format.finish_mark not exists for 8-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358329953831715 TiDB Cluster ID. tikv_gc_leader_uuid 657fee77cf0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:4256, start at 2025-04-04 15:35:26.297529139 +0800 CST m=+2.551508081 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:26.306 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:26.268 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:26.268 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358329953831715 TiDB Cluster ID. tikv_gc_leader_uuid 657fee77cf0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:4256, start at 2025-04-04 15:35:26.297529139 +0800 CST m=+2.551508081 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:26.306 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:26.268 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:26.268 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358330508763784 TiDB Cluster ID. tikv_gc_leader_uuid 657fee7802c0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:4332, start at 2025-04-04 15:35:26.478129889 +0800 CST m=+2.658427037 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:26.488 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:26.475 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:26.475 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:35:28 CST 2025] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.28742876.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server2 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.finish_mark not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:35:29 CST 2025] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.78197821.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 8.15 secs (494393194 bytes/sec) [Pipeline] { [Pipeline] cache The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting TLS PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 table test.finish_mark not exists for 4-th check, retry later table row_format.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting TLS TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:31 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/4661b051-81f5-45a5-85c9-0956a5b820b2 {UpstreamID:7489358203854966258 Namespace:default ID:4661b051-81f5-45a5-85c9-0956a5b820b2 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:35:04.296637471 +0800 CST StartTs:457114151593181197 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003775050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114151619395591} {CheckpointTs:457114156809584649 MinTableBarrierTs:457114157346979849 AdminJobType:noop} span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114156809584649, checkpointTs: 457114156809584649, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8375b438-860e-4220-99a5-a82351fde176 {"id":"8375b438-860e-4220-99a5-a82351fde176","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752128} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b2d17 8375b438-860e-4220-99a5-a82351fde176 /tidb/cdc/default/default/changefeed/info/4661b051-81f5-45a5-85c9-0956a5b820b2 {"upstream-id":7489358203854966258,"namespace":"default","changefeed-id":"4661b051-81f5-45a5-85c9-0956a5b820b2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:35:04.296637471+08:00","start-ts":457114151593181197,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114151619395591} /tidb/cdc/default/default/changefeed/status/4661b051-81f5-45a5-85c9-0956a5b820b2 {"checkpoint-ts":457114156809584649,"min-table-barrier-ts":457114157346979849,"admin-job-type":0} /tidb/cdc/default/default/task/position/8375b438-860e-4220-99a5-a82351fde176/4661b051-81f5-45a5-85c9-0956a5b820b2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358203854966258 {"id":7489358203854966258,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/4661b051-81f5-45a5-85c9-0956a5b820b2 {UpstreamID:7489358203854966258 Namespace:default ID:4661b051-81f5-45a5-85c9-0956a5b820b2 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:35:04.296637471 +0800 CST StartTs:457114151593181197 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003775050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114151619395591} {CheckpointTs:457114156809584649 MinTableBarrierTs:457114157346979849 AdminJobType:noop} span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114156809584649, checkpointTs: 457114156809584649, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8375b438-860e-4220-99a5-a82351fde176 {"id":"8375b438-860e-4220-99a5-a82351fde176","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752128} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b2d17 8375b438-860e-4220-99a5-a82351fde176 /tidb/cdc/default/default/changefeed/info/4661b051-81f5-45a5-85c9-0956a5b820b2 {"upstream-id":7489358203854966258,"namespace":"default","changefeed-id":"4661b051-81f5-45a5-85c9-0956a5b820b2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:35:04.296637471+08:00","start-ts":457114151593181197,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114151619395591} /tidb/cdc/default/default/changefeed/status/4661b051-81f5-45a5-85c9-0956a5b820b2 {"checkpoint-ts":457114156809584649,"min-table-barrier-ts":457114157346979849,"admin-job-type":0} /tidb/cdc/default/default/task/position/8375b438-860e-4220-99a5-a82351fde176/4661b051-81f5-45a5-85c9-0956a5b820b2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358203854966258 {"id":7489358203854966258,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/4661b051-81f5-45a5-85c9-0956a5b820b2 {UpstreamID:7489358203854966258 Namespace:default ID:4661b051-81f5-45a5-85c9-0956a5b820b2 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:35:04.296637471 +0800 CST StartTs:457114151593181197 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003775050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114151619395591} {CheckpointTs:457114156809584649 MinTableBarrierTs:457114157346979849 AdminJobType:noop} span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114156809584649, checkpointTs: 457114156809584649, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114156809584650, checkpointTs: 457114156809584649, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8375b438-860e-4220-99a5-a82351fde176 {"id":"8375b438-860e-4220-99a5-a82351fde176","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752128} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb94b2d17 8375b438-860e-4220-99a5-a82351fde176 /tidb/cdc/default/default/changefeed/info/4661b051-81f5-45a5-85c9-0956a5b820b2 {"upstream-id":7489358203854966258,"namespace":"default","changefeed-id":"4661b051-81f5-45a5-85c9-0956a5b820b2","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:35:04.296637471+08:00","start-ts":457114151593181197,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114151619395591} /tidb/cdc/default/default/changefeed/status/4661b051-81f5-45a5-85c9-0956a5b820b2 {"checkpoint-ts":457114156809584649,"min-table-barrier-ts":457114157346979849,"admin-job-type":0} /tidb/cdc/default/default/task/position/8375b438-860e-4220-99a5-a82351fde176/4661b051-81f5-45a5-85c9-0956a5b820b2 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358203854966258 + grep -q 'etcd info' {"id":7489358203854966258,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table test.finish_mark exists check table exists success check diff successfully table row_format.finish_mark not exists for 10-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:32 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfc90c47-2962-4152-bb29-52f1a27509ef {"id":"cfc90c47-2962-4152-bb29-52f1a27509ef","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc22d6 cfc90c47-2962-4152-bb29-52f1a27509ef /tidb/cdc/default/default/upstream/7489358323780607030 {"id":7489358323780607030,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfc90c47-2962-4152-bb29-52f1a27509ef {"id":"cfc90c47-2962-4152-bb29-52f1a27509ef","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc22d6 cfc90c47-2962-4152-bb29-52f1a27509ef /tidb/cdc/default/default/upstream/7489358323780607030 {"id":7489358323780607030,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cfc90c47-2962-4152-bb29-52f1a27509ef {"id":"cfc90c47-2962-4152-bb29-52f1a27509ef","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc22d6 cfc90c47-2962-4152-bb29-52f1a27509ef /tidb/cdc/default/default/upstream/7489358323780607030 {"id":7489358323780607030,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:35:32 CST 2025] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.78887890.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358352222110577 TiDB Cluster ID. tikv_gc_leader_uuid 657fee7c3740003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:4024, start at 2025-04-04 15:35:30.783554401 +0800 CST m=+2.072272555 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:30.791 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:30.831 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:30.831 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358352222110577 TiDB Cluster ID. tikv_gc_leader_uuid 657fee7c3740003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:4024, start at 2025-04-04 15:35:30.783554401 +0800 CST m=+2.072272555 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:30.791 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:30.831 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:30.831 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358355294867137 TiDB Cluster ID. tikv_gc_leader_uuid 657fee7c2a8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:4080, start at 2025-04-04 15:35:30.753703467 +0800 CST m=+1.982208045 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:30.760 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:30.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:30.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting TLS TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:35:33 CST 2025] <<<<<< run test case charset_gbk success! >>>>>> table row_format.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... [Fri Apr 4 15:35:35 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.53975399.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358372397244871 TiDB Cluster ID. tikv_gc_leader_uuid 657fee80580000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:5899, start at 2025-04-04 15:35:35.052585348 +0800 CST m=+1.694521751 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:35.063 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:35.058 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:35.058 +0800 All versions after safe point can be accessed. (DO NOT EDIT) wait process cdc.test exit for 2-th time... [Fri Apr 4 15:35:35 CST 2025] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.59365938.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls1 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:35 GMT < Content-Length: 1261 < Content-Type: text/plain; charset=utf-8 < { [1261 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/87ee4480-c5d2-4e7c-81db-f982b29b697c {"id":"87ee4480-c5d2-4e7c-81db-f982b29b697c","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752132} /tidb/cdc/default/__cdc_meta__/capture/cfc90c47-2962-4152-bb29-52f1a27509ef {"id":"cfc90c47-2962-4152-bb29-52f1a27509ef","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc22d6 cfc90c47-2962-4152-bb29-52f1a27509ef /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc2316 87ee4480-c5d2-4e7c-81db-f982b29b697c /tidb/cdc/default/default/upstream/7489358323780607030 {"id":7489358323780607030,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/87ee4480-c5d2-4e7c-81db-f982b29b697c {"id":"87ee4480-c5d2-4e7c-81db-f982b29b697c","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752132} /tidb/cdc/default/__cdc_meta__/capture/cfc90c47-2962-4152-bb29-52f1a27509ef {"id":"cfc90c47-2962-4152-bb29-52f1a27509ef","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc22d6 cfc90c47-2962-4152-bb29-52f1a27509ef /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc2316 87ee4480-c5d2-4e7c-81db-f982b29b697c /tidb/cdc/default/default/upstream/7489358323780607030 {"id":7489358323780607030,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/87ee4480-c5d2-4e7c-81db-f982b29b697c {"id":"87ee4480-c5d2-4e7c-81db-f982b29b697c","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752132} /tidb/cdc/default/__cdc_meta__/capture/cfc90c47-2962-4152-bb29-52f1a27509ef {"id":"cfc90c47-2962-4152-bb29-52f1a27509ef","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc22d6 cfc90c47-2962-4152-bb29-52f1a27509ef /tidb/cdc/default/__cdc_meta__/owner/223195ffb9bc2316 87ee4480-c5d2-4e7c-81db-f982b29b697c /tidb/cdc/default/default/upstream/7489358323780607030 {"id":7489358323780607030,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table capture_suicide_while_balance_table.t1 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:35:36 CST 2025] <<<<<< run test case row_format success! >>>>>> table capture_suicide_while_balance_table.t1 exists table capture_suicide_while_balance_table.t2 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:38 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17dfbecf-a49a-4d6d-b80b-310088d80c32 {"id":"17dfbecf-a49a-4d6d-b80b-310088d80c32","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9da93d7 17dfbecf-a49a-4d6d-b80b-310088d80c32 /tidb/cdc/default/default/upstream/7489358352222110577 {"id":7489358352222110577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17dfbecf-a49a-4d6d-b80b-310088d80c32 {"id":"17dfbecf-a49a-4d6d-b80b-310088d80c32","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9da93d7 17dfbecf-a49a-4d6d-b80b-310088d80c32 /tidb/cdc/default/default/upstream/7489358352222110577 {"id":7489358352222110577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/17dfbecf-a49a-4d6d-b80b-310088d80c32 {"id":"17dfbecf-a49a-4d6d-b80b-310088d80c32","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9da93d7 17dfbecf-a49a-4d6d-b80b-310088d80c32 /tidb/cdc/default/default/upstream/7489358352222110577 {"id":7489358352222110577,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:38 GMT < Content-Length: 1210 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/266782cf-ba63-410f-9beb-ec2bbd17facc {"id":"266782cf-ba63-410f-9beb-ec2bbd17facc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef6db 266782cf-ba63-410f-9beb-ec2bbd17facc /tidb/cdc/default/default/upstream/7489358372397244871 {"id":7489358372397244871,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/266782cf-ba63-410f-9beb-ec2bbd17facc {"id":"266782cf-ba63-410f-9beb-ec2bbd17facc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef6db 266782cf-ba63-410f-9beb-ec2bbd17facc /tidb/cdc/default/default/upstream/7489358372397244871 {"id":7489358372397244871,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/266782cf-ba63-410f-9beb-ec2bbd17facc {"id":"266782cf-ba63-410f-9beb-ec2bbd17facc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef6db 266782cf-ba63-410f-9beb-ec2bbd17facc /tidb/cdc/default/default/upstream/7489358372397244871 {"id":7489358372397244871,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "updateproportion"="0" "operationcount"="0" "threadcount"="4" "workload"="core" "readproportion"="0" "scanproportion"="0" "recordcount"="50" "insertproportion"="0" "mysql.db"="changefeed_reconstruct" "requestdistribution"="uniform" "dotransactions"="false" "mysql.port"="4000" "mysql.host"="127.0.0.1" "mysql.user"="root" "readallfields"="true" ********************************************** Run finished, takes 20.513257ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3554.5, Avg(us): 1629, Min(us): 873, Max(us): 6979, 95th(us): 7000, 99th(us): 7000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later table capture_suicide_while_balance_table.t2 exists table capture_suicide_while_balance_table.t3 not exists for 1-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 8.15 secs (494581886 bytes/sec) [Pipeline] { [Pipeline] cache [Fri Apr 4 15:35:40 CST 2025] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.60096011.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls2 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8301 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later table capture_suicide_while_balance_table.t3 exists table capture_suicide_while_balance_table.t4 not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:35:42 CST 2025] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table capture_suicide_while_balance_table.t4 exists + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:43 GMT < Content-Length: 1661 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/266782cf-ba63-410f-9beb-ec2bbd17facc {"id":"266782cf-ba63-410f-9beb-ec2bbd17facc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/capture/68ecd556-6ac9-47b8-8d58-518e78897aad {"id":"68ecd556-6ac9-47b8-8d58-518e78897aad","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752140} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef6db 266782cf-ba63-410f-9beb-ec2bbd17facc /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef718 68ecd556-6ac9-47b8-8d58-518e78897aad /tidb/cdc/default/default/upstream/7489358372397244871 {"id":7489358372397244871,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/266782cf-ba63-410f-9beb-ec2bbd17facc {"id":"266782cf-ba63-410f-9beb-ec2bbd17facc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/capture/68ecd556-6ac9-47b8-8d58-518e78897aad {"id":"68ecd556-6ac9-47b8-8d58-518e78897aad","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752140} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef6db 266782cf-ba63-410f-9beb-ec2bbd17facc /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef718 68ecd556-6ac9-47b8-8d58-518e78897aad /tidb/cdc/default/default/upstream/7489358372397244871 {"id":7489358372397244871,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/266782cf-ba63-410f-9beb-ec2bbd17facc {"id":"266782cf-ba63-410f-9beb-ec2bbd17facc","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752135} /tidb/cdc/default/__cdc_meta__/capture/68ecd556-6ac9-47b8-8d58-518e78897aad {"id":"68ecd556-6ac9-47b8-8d58-518e78897aad","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752140} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef6db 266782cf-ba63-410f-9beb-ec2bbd17facc /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffb9eef718 68ecd556-6ac9-47b8-8d58-518e78897aad /tidb/cdc/default/default/upstream/7489358372397244871 {"id":7489358372397244871,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x wait process 5402 exit for 1-th time... wait process 5402 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5402) - No such process wait process 5402 exit for 3-th time... process 5402 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:35:45 CST 2025] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.56605662.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 pass test: check health pass test: get status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 140 100 35 100 105 1060 3181 --:--:-- --:--:-- --:--:-- 4242 { "status": true, "message": "" pass test: create changefeed check_changefeed_state https://127.0.0.1:2579 changefeed-test1 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test1 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test1 -s + info='{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114162602704906, "checkpoint_time": "2025-04-04 15:35:46.159", "error": null }' + echo '{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114162602704906, "checkpoint_time": "2025-04-04 15:35:46.159", "error": null }' { "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114162602704906, "checkpoint_time": "2025-04-04 15:35:46.159", "error": null } ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114162602704906, '"checkpoint_time":' '"2025-04-04' '15:35:46.159",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114162602704906, '"checkpoint_time":' '"2025-04-04' '15:35:46.159",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test2 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test2 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test2 -s start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + info='{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114162642026503, "checkpoint_time": "2025-04-04 15:35:46.309", "error": null }' + echo '{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114162642026503, "checkpoint_time": "2025-04-04 15:35:46.309", "error": null }' { "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114162642026503, "checkpoint_time": "2025-04-04 15:35:46.309", "error": null } ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114162642026503, '"checkpoint_time":' '"2025-04-04' '15:35:46.309",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114162642026503, '"checkpoint_time":' '"2025-04-04' '15:35:46.309",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test3 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test3 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test3 -s + info='{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114162667978763, "checkpoint_time": "2025-04-04 15:35:46.408", "error": null }' + echo '{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114162667978763, "checkpoint_time": "2025-04-04 15:35:46.408", "error": null }' { "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114162667978763, "checkpoint_time": "2025-04-04 15:35:46.408", "error": null } ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114162667978763, '"checkpoint_time":' '"2025-04-04' '15:35:46.408",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114162667978763, '"checkpoint_time":' '"2025-04-04' '15:35:46.408",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test4 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test4 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test4 -s + info='{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114162930384903, "checkpoint_time": "2025-04-04 15:35:47.409", "error": null }' + echo '{ "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114162930384903, "checkpoint_time": "2025-04-04 15:35:47.409", "error": null }' { "upstream_id": 7489358372397244871, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114162930384903, "checkpoint_time": "2025-04-04 15:35:47.409", "error": null } ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114162930384903, '"checkpoint_time":' '"2025-04-04' '15:35:47.409",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358372397244871, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114162930384903, '"checkpoint_time":' '"2025-04-04' '15:35:47.409",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:35:48 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [4090 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {UpstreamID:7489358352222110577 Namespace:default ID:24fab03b-26c1-4a1d-b856-d1c03aeb3559 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:35:38.360727422 +0800 CST StartTs:457114160524427269 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e613b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114160550641671} {CheckpointTs:457114163014795276 MinTableBarrierTs:457114163027902465 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114163014795276, checkpointTs: 457114163014795276, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e757d76-82be-47ad-9a90-0a7db5e81b5e {"id":"2e757d76-82be-47ad-9a90-0a7db5e81b5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752145} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9da9480 2e757d76-82be-47ad-9a90-0a7db5e81b5e /tidb/cdc/default/default/changefeed/info/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"upstream-id":7489358352222110577,"namespace":"default","changefeed-id":"24fab03b-26c1-4a1d-b856-d1c03aeb3559","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:35:38.360727422+08:00","start-ts":457114160524427269,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114160550641671} /tidb/cdc/default/default/changefeed/status/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"checkpoint-ts":457114163014795276,"min-table-barrier-ts":457114163027902465,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e757d76-82be-47ad-9a90-0a7db5e81b5e/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358352222110577 {"id":7489358352222110577,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {UpstreamID:7489358352222110577 Namespace:default ID:24fab03b-26c1-4a1d-b856-d1c03aeb3559 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:35:38.360727422 +0800 CST StartTs:457114160524427269 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e613b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114160550641671} {CheckpointTs:457114163014795276 MinTableBarrierTs:457114163027902465 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114163014795276, checkpointTs: 457114163014795276, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e757d76-82be-47ad-9a90-0a7db5e81b5e {"id":"2e757d76-82be-47ad-9a90-0a7db5e81b5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752145} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9da9480 2e757d76-82be-47ad-9a90-0a7db5e81b5e /tidb/cdc/default/default/changefeed/info/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"upstream-id":7489358352222110577,"namespace":"default","changefeed-id":"24fab03b-26c1-4a1d-b856-d1c03aeb3559","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:35:38.360727422+08:00","start-ts":457114160524427269,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114160550641671} /tidb/cdc/default/default/changefeed/status/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"checkpoint-ts":457114163014795276,"min-table-barrier-ts":457114163027902465,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e757d76-82be-47ad-9a90-0a7db5e81b5e/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358352222110577 {"id":7489358352222110577,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {UpstreamID:7489358352222110577 Namespace:default ID:24fab03b-26c1-4a1d-b856-d1c03aeb3559 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:35:38.360727422 +0800 CST StartTs:457114160524427269 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e613b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114160550641671} {CheckpointTs:457114163014795276 MinTableBarrierTs:457114163027902465 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114163014795276, checkpointTs: 457114163014795276, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e757d76-82be-47ad-9a90-0a7db5e81b5e {"id":"2e757d76-82be-47ad-9a90-0a7db5e81b5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752145} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffb9da9480 2e757d76-82be-47ad-9a90-0a7db5e81b5e /tidb/cdc/default/default/changefeed/info/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"upstream-id":7489358352222110577,"namespace":"default","changefeed-id":"24fab03b-26c1-4a1d-b856-d1c03aeb3559","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:35:38.360727422+08:00","start-ts":457114160524427269,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114160550641671} /tidb/cdc/default/default/changefeed/status/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"checkpoint-ts":457114163014795276,"min-table-barrier-ts":457114163027902465,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e757d76-82be-47ad-9a90-0a7db5e81b5e/24fab03b-26c1-4a1d-b856-d1c03aeb3559 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358352222110577 {"id":7489358352222110577,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id }lease 223195ffb9bc2316 revoked check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 6.88 secs (586183987 bytes/sec) [Pipeline] { [Pipeline] cache "id": "2e757d76-82be-47ad-9a90-0a7db5e81b5e", "cluster-id": "default" run task successfully table test.verify_table_eligible not exists for 1-th check, retry later check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "mysql.db"="sink_retry" "operationcount"="0" "updateproportion"="0" "readproportion"="0" "recordcount"="10" "mysql.host"="127.0.0.1" "dotransactions"="false" "readallfields"="true" "requestdistribution"="uniform" "threadcount"="2" "mysql.user"="root" "insertproportion"="0" "mysql.port"="4000" "scanproportion"="0" "workload"="core" ********************************************** Run finished, takes 4.437063ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3340.9, Avg(us): 815, Min(us): 521, Max(us): 1883, 95th(us): 2000, 99th(us): 2000 start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... capture_id: 2e757d76-82be-47ad-9a90-0a7db5e81b5e check_processor_table_count http://127.0.0.1:2379 24fab03b-26c1-4a1d-b856-d1c03aeb3559 2e757d76-82be-47ad-9a90-0a7db5e81b5e 1 run task successfully table test.verify_table_eligible not exists for 2-th check, retry later check_processor_table_count http://127.0.0.1:2379 24fab03b-26c1-4a1d-b856-d1c03aeb3559 2e757d76-82be-47ad-9a90-0a7db5e81b5e 0 table count 1 does equal to expected count 0 run task failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table test.verify_table_eligible exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: mysql... <<================= pass test: list changefeed Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore check_processor_table_count http://127.0.0.1:2379 24fab03b-26c1-4a1d-b856-d1c03aeb3559 2e757d76-82be-47ad-9a90-0a7db5e81b5e 0 run task successfully Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358446377843133 TiDB Cluster ID. tikv_gc_leader_uuid 657fee9288c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:3758, start at 2025-04-04 15:35:53.666998048 +0800 CST m=+2.767630569 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:53.675 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:53.635 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:53.635 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358446377843133 TiDB Cluster ID. tikv_gc_leader_uuid 657fee9288c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:3758, start at 2025-04-04 15:35:53.666998048 +0800 CST m=+2.767630569 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:53.675 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:53.635 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:53.635 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358451034477700 TiDB Cluster ID. tikv_gc_leader_uuid 657fee92938000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:3843, start at 2025-04-04 15:35:53.715243683 +0800 CST m=+2.726167548 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:53.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:53.729 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:53.729 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... ***************** properties ***************** "readallfields"="true" "readproportion"="0" "recordcount"="50" "dotransactions"="false" "scanproportion"="0" "threadcount"="4" "mysql.user"="root" "operationcount"="0" "workload"="core" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "mysql.db"="changefeed_reconstruct" "updateproportion"="0" "requestdistribution"="uniform" ********************************************** Run finished, takes 16.661835ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3929.6, Avg(us): 1288, Min(us): 859, Max(us): 4323, 95th(us): 5000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... pass test: get changefeed wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:35:57 CST 2025] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> [Fri Apr 4 15:35:57 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.52225224.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later pass test: pause changefeed VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358463012425416 TiDB Cluster ID. tikv_gc_leader_uuid 657fee9591c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:3986, start at 2025-04-04 15:35:56.79004046 +0800 CST m=+1.964075033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:56.828 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:56.826 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:56.826 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358463012425416 TiDB Cluster ID. tikv_gc_leader_uuid 657fee9591c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:3986, start at 2025-04-04 15:35:56.79004046 +0800 CST m=+1.964075033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:56.828 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:56.826 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:56.826 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358464934045519 TiDB Cluster ID. tikv_gc_leader_uuid 657fee95cf80009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:4047, start at 2025-04-04 15:35:57.008710966 +0800 CST m=+2.119890084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:37:57.033 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:35:57.040 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:25:57.040 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.88 secs (511643404 bytes/sec) [Pipeline] { [Pipeline] cache pass test: update changefeed check diff successfully wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:00 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ddd669cd-904c-4d62-86e7-7e583cf3009c {"id":"ddd669cd-904c-4d62-86e7-7e583cf3009c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752157} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba3143d7 ddd669cd-904c-4d62-86e7-7e583cf3009c /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ddd669cd-904c-4d62-86e7-7e583cf3009c {"id":"ddd669cd-904c-4d62-86e7-7e583cf3009c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752157} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba3143d7 ddd669cd-904c-4d62-86e7-7e583cf3009c /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ddd669cd-904c-4d62-86e7-7e583cf3009c {"id":"ddd669cd-904c-4d62-86e7-7e583cf3009c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752157} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba3143d7 ddd669cd-904c-4d62-86e7-7e583cf3009c /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.5279.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-manager Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7489358446377843133,"namespace":"default","id":"ddl-manager","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:36:00.865889302+08:00","start_ts":457114166436560897,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114166436560897,"checkpoint_ts":457114166436560897,"checkpoint_time":"2025-04-04 15:36:00.784"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5281.out cli tso query --pd=http://127.0.0.1:2379 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:36:01 CST 2025] <<<<<< run test case changefeed_reconstruct success! >>>>>> + set +x start tidb cluster in /tmp/tidb_cdc_test/cdc Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... pass test: resume changefeed + set +x + tso='457114166648635393 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114166648635393 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:36:03 CST 2025] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.53235325.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release pass test: rebalance table wait process 5227 exit for 1-th time... wait process 5227 exit for 2-th time... wait process 5227 exit for 3-th time... pass test: list processors wait process 5227 exit for 4-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 5227 exit for 5-th time... wait process 5227 exit for 6-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:06 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff848d48-2ec5-4522-95c9-028a1a1c9dc4 {"id":"ff848d48-2ec5-4522-95c9-028a1a1c9dc4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752163} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba4093e4 ff848d48-2ec5-4522-95c9-028a1a1c9dc4 /tidb/cdc/default/default/upstream/7489358463012425416 {"id":7489358463012425416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff848d48-2ec5-4522-95c9-028a1a1c9dc4 {"id":"ff848d48-2ec5-4522-95c9-028a1a1c9dc4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752163} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba4093e4 ff848d48-2ec5-4522-95c9-028a1a1c9dc4 /tidb/cdc/default/default/upstream/7489358463012425416 {"id":7489358463012425416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff848d48-2ec5-4522-95c9-028a1a1c9dc4 {"id":"ff848d48-2ec5-4522-95c9-028a1a1c9dc4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752163} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba4093e4 ff848d48-2ec5-4522-95c9-028a1a1c9dc4 /tidb/cdc/default/default/upstream/7489358463012425416 {"id":7489358463012425416,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5377.out cli changefeed create --start-ts=457114166648635393 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: a9c53080-7c36-4493-91a3-6e9a362c0366 Info: {"upstream_id":7489358463012425416,"namespace":"default","id":"a9c53080-7c36-4493-91a3-6e9a362c0366","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:36:06.576746291+08:00","start_ts":457114166648635393,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114166648635393,"checkpoint_ts":457114166648635393,"checkpoint_time":"2025-04-04 15:36:01.593"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process 5227 exit for 7-th time... wait process 5227 exit for 8-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x wait process 5227 exit for 9-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.04 secs (572267350 bytes/sec) pass test: get processor [Pipeline] { [Pipeline] cache wait process 5227 exit for 10-th time... wait process 5227 exit for 11-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5227) - No such process wait process 5227 exit for 12-th time... process 5227 already exit [Fri Apr 4 15:36:09 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.53565358.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358510386947579 TiDB Cluster ID. tikv_gc_leader_uuid 657feea0014000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:4042, start at 2025-04-04 15:36:07.452186715 +0800 CST m=+2.199235876 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:07.459 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:07.429 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:07.429 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358510386947579 TiDB Cluster ID. tikv_gc_leader_uuid 657feea0014000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:4042, start at 2025-04-04 15:36:07.452186715 +0800 CST m=+2.199235876 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:07.459 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:07.429 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:07.429 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358510937771949 TiDB Cluster ID. tikv_gc_leader_uuid 657feea019c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:4119, start at 2025-04-04 15:36:07.540114338 +0800 CST m=+2.195371653 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:07.547 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:07.527 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:07.527 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cdc/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } pass test: move table pass test: set log level [Fri Apr 4 15:36:11 CST 2025] <<<<<< START cdc server in cdc case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.54915493.out server --log-file /tmp/tidb_cdc_test/cdc/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table drop_tables.c not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/syncpoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3696 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489358446377843133 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:36:00.865889302 +0800 CST StartTs:457114166436560897 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041c0f30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114166449930246} {CheckpointTs:457114166842884119 MinTableBarrierTs:457114166842884119 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/726b287c-72cb-44e5-a1d1-5c18a22b9c4a {"id":"726b287c-72cb-44e5-a1d1-5c18a22b9c4a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752169} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba31459f 726b287c-72cb-44e5-a1d1-5c18a22b9c4a /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489358446377843133,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:36:00.865889302+08:00","start-ts":457114166436560897,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114166449930246} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114166856253444,"min-table-barrier-ts":457114166856253444,"admin-job-type":0} /tidb/cdc/default/default/task/position/726b287c-72cb-44e5-a1d1-5c18a22b9c4a/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489358446377843133 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:36:00.865889302 +0800 CST StartTs:457114166436560897 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041c0f30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114166449930246} {CheckpointTs:457114166842884119 MinTableBarrierTs:457114166842884119 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/726b287c-72cb-44e5-a1d1-5c18a22b9c4a {"id":"726b287c-72cb-44e5-a1d1-5c18a22b9c4a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752169} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba31459f 726b287c-72cb-44e5-a1d1-5c18a22b9c4a /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489358446377843133,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:36:00.865889302+08:00","start-ts":457114166436560897,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114166449930246} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114166856253444,"min-table-barrier-ts":457114166856253444,"admin-job-type":0} /tidb/cdc/default/default/task/position/726b287c-72cb-44e5-a1d1-5c18a22b9c4a/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489358446377843133 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:36:00.865889302 +0800 CST StartTs:457114166436560897 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041c0f30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114166449930246} {CheckpointTs:457114166842884119 MinTableBarrierTs:457114166842884119 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/726b287c-72cb-44e5-a1d1-5c18a22b9c4a {"id":"726b287c-72cb-44e5-a1d1-5c18a22b9c4a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752169} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba31459f 726b287c-72cb-44e5-a1d1-5c18a22b9c4a /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489358446377843133,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:36:00.865889302+08:00","start-ts":457114166436560897,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114166449930246} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114166856253444,"min-table-barrier-ts":457114166856253444,"admin-job-type":0} /tidb/cdc/default/default/task/position/726b287c-72cb-44e5-a1d1-5c18a22b9c4a/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:36:12 CST 2025] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.54195421.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > pass test: remove changefeed < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:13 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3696 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489358446377843133 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:36:00.865889302 +0800 CST StartTs:457114166436560897 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041c0f30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114166449930246} {CheckpointTs:457114166856253444 MinTableBarrierTs:457114166856253444 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/726b287c-72cb-44e5-a1d1-5c18a22b9c4a {"id":"726b287c-72cb-44e5-a1d1-5c18a22b9c4a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752169} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba31459f 726b287c-72cb-44e5-a1d1-5c18a22b9c4a /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489358446377843133,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:36:00.865889302+08:00","start-ts":457114166436560897,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114166449930246} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114166856253444,"min-table-barrier-ts":457114166856253444,"admin-job-type":0} /tidb/cdc/default/default/task/position/726b287c-72cb-44e5-a1d1-5c18a22b9c4a/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489358446377843133 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:36:00.865889302 +0800 CST StartTs:457114166436560897 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041c0f30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114166449930246} {CheckpointTs:457114166856253444 MinTableBarrierTs:457114166856253444 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/726b287c-72cb-44e5-a1d1-5c18a22b9c4a {"id":"726b287c-72cb-44e5-a1d1-5c18a22b9c4a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752169} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba31459f 726b287c-72cb-44e5-a1d1-5c18a22b9c4a /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489358446377843133,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:36:00.865889302+08:00","start-ts":457114166436560897,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114166449930246} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114166856253444,"min-table-barrier-ts":457114166856253444,"admin-job-type":0} /tidb/cdc/default/default/task/position/726b287c-72cb-44e5-a1d1-5c18a22b9c4a/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7489358446377843133 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:36:00.865889302 +0800 CST StartTs:457114166436560897 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041c0f30 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114166449930246} {CheckpointTs:457114166856253444 MinTableBarrierTs:457114166856253444 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/726b287c-72cb-44e5-a1d1-5c18a22b9c4a {"id":"726b287c-72cb-44e5-a1d1-5c18a22b9c4a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752169} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba31459f 726b287c-72cb-44e5-a1d1-5c18a22b9c4a /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7489358446377843133,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:36:00.865889302+08:00","start-ts":457114166436560897,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114166449930246} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":457114166856253444,"min-table-barrier-ts":457114166856253444,"admin-job-type":0} /tidb/cdc/default/default/task/position/726b287c-72cb-44e5-a1d1-5c18a22b9c4a/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358446377843133 {"id":7489358446377843133,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later pass test: resign owner =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table drop_tables.c not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:14 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59e70742-d3f3-4881-9d04-7d88c3f22de5 {"id":"59e70742-d3f3-4881-9d04-7d88c3f22de5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752172} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba6959d8 59e70742-d3f3-4881-9d04-7d88c3f22de5 /tidb/cdc/default/default/upstream/7489358510386947579 {"id":7489358510386947579,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59e70742-d3f3-4881-9d04-7d88c3f22de5 {"id":"59e70742-d3f3-4881-9d04-7d88c3f22de5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752172} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba6959d8 59e70742-d3f3-4881-9d04-7d88c3f22de5 /tidb/cdc/default/default/upstream/7489358510386947579 {"id":7489358510386947579,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/59e70742-d3f3-4881-9d04-7d88c3f22de5 {"id":"59e70742-d3f3-4881-9d04-7d88c3f22de5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752172} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba6959d8 59e70742-d3f3-4881-9d04-7d88c3f22de5 /tidb/cdc/default/default/upstream/7489358510386947579 {"id":7489358510386947579,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.cli.5556.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config /tmp/tidb_cdc_test/cdc/pulsar_test.toml pass test: get tso Create changefeed successfully! ID: 2e3803d4-6f42-4177-94c6-60ed4099264b Info: {"upstream_id":7489358510386947579,"namespace":"default","id":"2e3803d4-6f42-4177-94c6-60ed4099264b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:36:15.209708847+08:00","start_ts":457114170184171526,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114170184171526,"checkpoint_ts":457114170184171526,"checkpoint_time":"2025-04-04 15:36:15.080"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release pass test: verify table table drop_tables.c not exists for 3-th check, retry later + set +x go: downloading go1.23.4 (linux/amd64) start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: create changefeed v2 table drop_tables.c exists check diff successfully Verifying downstream PD is started... wait process cdc.test exit for 1-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 8.33 secs (483805203 bytes/sec) [Pipeline] { [Pipeline] cache pass test: remove changefeed v2 wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 4-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:36:19 CST 2025] <<<<<< run test case drop_many_tables success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358559875417447 TiDB Cluster ID. tikv_gc_leader_uuid 657feeabea80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:9262, start at 2025-04-04 15:36:19.628044772 +0800 CST m=+2.023679389 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:19.635 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:19.626 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:19.626 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358559875417447 TiDB Cluster ID. tikv_gc_leader_uuid 657feeabea80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:9262, start at 2025-04-04 15:36:19.628044772 +0800 CST m=+2.023679389 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:19.635 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:19.626 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:19.626 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) status code 200 pass test: show metadata status code 200 status code 200 pass test: delete service_gc_safepoint status code 500 pass test: resolve lock Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 5-th check, retry later wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358561309518408 TiDB Cluster ID. tikv_gc_leader_uuid 657feeac5ac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:9327, start at 2025-04-04 15:36:20.119768603 +0800 CST m=+2.450436908 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:20.132 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:20.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:20.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/syncpoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/syncpoint/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:36:21 CST 2025] <<<<<< run test case sink_retry success! >>>>>> go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:36:23 CST 2025] <<<<<< run test case http_api_tls success! >>>>>> table ddl_manager.finish_mark not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading golang.org/x/text v0.21.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/spf13/pflag v1.0.5 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading golang.org/x/tools v0.28.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading golang.org/x/sys v0.28.0 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.10714.out cli tso query --pd=http://127.0.0.1:2379 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading cloud.google.com/go v0.112.2 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/golang/snappy v0.0.4 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 table ddl_manager.finish_mark not exists for 7-th check, retry later go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible + set +x + tso='457114172777037825 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114172777037825 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:36:26 CST 2025] <<<<<< START cdc server in syncpoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/DDLSinkInitializeSlowly=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.1075010752.out server --log-file /tmp/tidb_cdc_test/syncpoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/syncpoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358574315710978 TiDB Cluster ID. tikv_gc_leader_uuid 657feeb0b6c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:6964, start at 2025-04-04 15:36:24.580946512 +0800 CST m=+2.135905273 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:24.626 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:24.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:24.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358574315710978 TiDB Cluster ID. tikv_gc_leader_uuid 657feeb0b6c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:6964, start at 2025-04-04 15:36:24.580946512 +0800 CST m=+2.135905273 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:24.626 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:24.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:24.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358582662524911 TiDB Cluster ID. tikv_gc_leader_uuid 657feeb0e4c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:7030, start at 2025-04-04 15:36:24.734052544 +0800 CST m=+2.195982562 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:24.741 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:24.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:24.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/kr/text v0.2.0 go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 table ddl_manager.finish_mark not exists for 8-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 6.87 secs (586482904 bytes/sec) [Pipeline] { [Pipeline] cache + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8339.out cli tso query --pd=http://127.0.0.1:2379 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible table ddl_manager.finish_mark not exists for 9-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:29 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/91f70ede-ed70-47c9-8136-3eb64509e3ca {"id":"91f70ede-ed70-47c9-8136-3eb64509e3ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752186} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba93b1f5 91f70ede-ed70-47c9-8136-3eb64509e3ca /tidb/cdc/default/default/upstream/7489358559875417447 {"id":7489358559875417447,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/91f70ede-ed70-47c9-8136-3eb64509e3ca {"id":"91f70ede-ed70-47c9-8136-3eb64509e3ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752186} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba93b1f5 91f70ede-ed70-47c9-8136-3eb64509e3ca /tidb/cdc/default/default/upstream/7489358559875417447 {"id":7489358559875417447,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/91f70ede-ed70-47c9-8136-3eb64509e3ca {"id":"91f70ede-ed70-47c9-8136-3eb64509e3ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752186} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffba93b1f5 91f70ede-ed70-47c9-8136-3eb64509e3ca /tidb/cdc/default/default/upstream/7489358559875417447 {"id":7489358559875417447,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.10812.out cli changefeed create --start-ts=457114172777037825 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint/conf/changefeed.toml Create changefeed successfully! ID: 3fcfdc87-99cb-4631-b66d-72389b978697 Info: {"upstream_id":7489358559875417447,"namespace":"default","id":"3fcfdc87-99cb-4631-b66d-72389b978697","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:36:30.076086824+08:00","start_ts":457114172777037825,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114172777037825,"checkpoint_ts":457114172777037825,"checkpoint_time":"2025-04-04 15:36:24.971"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114173869654018 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114173869654018 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "workload"="core" "recordcount"="10" "operationcount"="0" "mysql.db"="multi_capture_1" "dotransactions"="false" "updateproportion"="0" "threadcount"="2" "scanproportion"="0" "readallfields"="true" "readproportion"="0" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "mysql.port"="4000" "mysql.user"="root" "insertproportion"="0" ********************************************** Run finished, takes 10.983801ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1532.7, Avg(us): 1952, Min(us): 976, Max(us): 5880, 95th(us): 6000, 99th(us): 6000 ***************** properties ***************** "dotransactions"="false" "updateproportion"="0" "insertproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "readallfields"="true" "threadcount"="2" "requestdistribution"="uniform" "workload"="core" "mysql.port"="4000" "recordcount"="10" "mysql.user"="root" "mysql.db"="multi_capture_2" "scanproportion"="0" "readproportion"="0" ********************************************** Run finished, takes 8.848971ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1796.1, Avg(us): 1691, Min(us): 1152, Max(us): 3207, 95th(us): 4000, 99th(us): 4000 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 ***************** properties ***************** "scanproportion"="0" "mysql.host"="127.0.0.1" "recordcount"="10" "mysql.user"="root" "dotransactions"="false" "threadcount"="2" "updateproportion"="0" "workload"="core" "readallfields"="true" "mysql.db"="multi_capture_3" "requestdistribution"="uniform" "readproportion"="0" "insertproportion"="0" "operationcount"="0" "mysql.port"="4000" ********************************************** Run finished, takes 7.9265ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2093.6, Avg(us): 1497, Min(us): 962, Max(us): 3081, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "requestdistribution"="uniform" "insertproportion"="0" "mysql.db"="multi_capture_4" "readallfields"="true" "mysql.host"="127.0.0.1" "operationcount"="0" "readproportion"="0" "mysql.user"="root" "workload"="core" "updateproportion"="0" "recordcount"="10" "mysql.port"="4000" "dotransactions"="false" "scanproportion"="0" "threadcount"="2" ********************************************** Run finished, takes 7.680818ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2206.3, Avg(us): 1434, Min(us): 898, Max(us): 3019, 95th(us): 4000, 99th(us): 4000 [Fri Apr 4 15:36:30 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.84538455.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/jmespath/go-jmespath v0.4.0 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 10-th check, retry later + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ***************** properties ***************** "workload"="core" "mysql.db"="testSync" "mysql.host"="127.0.0.1" "insertproportion"="0" "dotransactions"="false" "requestdistribution"="uniform" "operationcount"="0" "readproportion"="0" "mysql.port"="4000" "scanproportion"="0" "recordcount"="10" "updateproportion"="0" "threadcount"="2" "mysql.user"="root" "readallfields"="true" ********************************************** Run finished, takes 8.961482ms INSERT - Takes(s): 0.0, Count: 9, OPS: 1813.5, Avg(us): 1442, Min(us): 1009, Max(us): 3833, 95th(us): 4000, 99th(us): 4000 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:34 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:36:34 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.85138515.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 12-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 6.87 secs (586873636 bytes/sec) start tidb cluster in /tmp/tidb_cdc_test/api_v2 Starting Upstream PD... [Pipeline] { [Pipeline] cache Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:37 GMT < Content-Length: 1261 < Content-Type: text/plain; charset=utf-8 < { [1261 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78421b9f-d79c-41b3-b0f9-69cc4da3775b {"id":"78421b9f-d79c-41b3-b0f9-69cc4da3775b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752194} /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4905d 78421b9f-d79c-41b3-b0f9-69cc4da3775b /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78421b9f-d79c-41b3-b0f9-69cc4da3775b {"id":"78421b9f-d79c-41b3-b0f9-69cc4da3775b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752194} /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4905d 78421b9f-d79c-41b3-b0f9-69cc4da3775b /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78421b9f-d79c-41b3-b0f9-69cc4da3775b {"id":"78421b9f-d79c-41b3-b0f9-69cc4da3775b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752194} /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4905d 78421b9f-d79c-41b3-b0f9-69cc4da3775b /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:36:37 CST 2025] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.85748576.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8303 failed: Connection refused * Failed to connect to 127.0.0.1 port 8303: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 13-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 14-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8303 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:40 GMT < Content-Length: 1735 < Content-Type: text/plain; charset=utf-8 < { [1735 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78421b9f-d79c-41b3-b0f9-69cc4da3775b {"id":"78421b9f-d79c-41b3-b0f9-69cc4da3775b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752194} /tidb/cdc/default/__cdc_meta__/capture/a6ef137a-3deb-40c3-aaee-f1472296caef {"id":"a6ef137a-3deb-40c3-aaee-f1472296caef","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752197} /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4905d 78421b9f-d79c-41b3-b0f9-69cc4da3775b /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4907e a6ef137a-3deb-40c3-aaee-f1472296caef /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78421b9f-d79c-41b3-b0f9-69cc4da3775b {"id":"78421b9f-d79c-41b3-b0f9-69cc4da3775b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752194} /tidb/cdc/default/__cdc_meta__/capture/a6ef137a-3deb-40c3-aaee-f1472296caef {"id":"a6ef137a-3deb-40c3-aaee-f1472296caef","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752197} /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4905d 78421b9f-d79c-41b3-b0f9-69cc4da3775b /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4907e a6ef137a-3deb-40c3-aaee-f1472296caef /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78421b9f-d79c-41b3-b0f9-69cc4da3775b {"id":"78421b9f-d79c-41b3-b0f9-69cc4da3775b","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752194} /tidb/cdc/default/__cdc_meta__/capture/a6ef137a-3deb-40c3-aaee-f1472296caef {"id":"a6ef137a-3deb-40c3-aaee-f1472296caef","address":"127.0.0.1:8303","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752197} /tidb/cdc/default/__cdc_meta__/capture/bbd7edc8-ceec-4dfe-951c-b8b979b86c21 {"id":"bbd7edc8-ceec-4dfe-951c-b8b979b86c21","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752191} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa49027 bbd7edc8-ceec-4dfe-951c-b8b979b86c21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4905d 78421b9f-d79c-41b3-b0f9-69cc4da3775b /tidb/cdc/default/__cdc_meta__/owner/223195ffbaa4907e a6ef137a-3deb-40c3-aaee-f1472296caef /tidb/cdc/default/default/upstream/7489358574315710978 {"id":7489358574315710978,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8627.out cli changefeed create --start-ts=457114173869654018 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 Create changefeed successfully! ID: a09df354-588d-4412-b02c-c67d9f098e48 Info: {"upstream_id":7489358574315710978,"namespace":"default","id":"a09df354-588d-4412-b02c-c67d9f098e48","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:36:40.607474963+08:00","start_ts":457114173869654018,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114173869654018,"checkpoint_ts":457114173869654018,"checkpoint_time":"2025-04-04 15:36:29.139"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 15-th check, retry later + set +x table multi_capture_1.usertable not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 16-th check, retry later table multi_capture_1.usertable exists table multi_capture_2.usertable not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358659000709430 TiDB Cluster ID. tikv_gc_leader_uuid 657feec33040014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:7301, start at 2025-04-04 15:36:43.495556895 +0800 CST m=+2.675869026 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:43.504 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:43.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:43.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358659000709430 TiDB Cluster ID. tikv_gc_leader_uuid 657feec33040014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:7301, start at 2025-04-04 15:36:43.495556895 +0800 CST m=+2.675869026 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:43.504 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:43.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:43.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358659943347538 TiDB Cluster ID. tikv_gc_leader_uuid 657feec36b00007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:7379, start at 2025-04-04 15:36:43.703297278 +0800 CST m=+2.805596772 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:43.716 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:43.692 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:43.692 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358661568166943 TiDB Cluster ID. tikv_gc_leader_uuid 657feec2f500005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:3884, start at 2025-04-04 15:36:43.266470347 +0800 CST m=+2.720413675 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:43.276 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:43.269 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:43.269 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358661568166943 TiDB Cluster ID. tikv_gc_leader_uuid 657feec2f500005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:3884, start at 2025-04-04 15:36:43.266470347 +0800 CST m=+2.720413675 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:43.276 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:43.269 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:43.269 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358653963502731 TiDB Cluster ID. tikv_gc_leader_uuid 657feec3190000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:3961, start at 2025-04-04 15:36:43.410420566 +0800 CST m=+2.790610881 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:43.419 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:43.414 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:43.414 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/api_v2/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/api_v2/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 17-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.70 secs (523395066 bytes/sec) [Pipeline] { [Pipeline] cache table multi_capture_2.usertable exists table multi_capture_3.usertable not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5323.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358659426718934 TiDB Cluster ID. tikv_gc_leader_uuid 657feec4dc40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:6461, start at 2025-04-04 15:36:45.175683569 +0800 CST m=+4.234188474 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:45.182 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:45.169 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:45.169 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358659426718934 TiDB Cluster ID. tikv_gc_leader_uuid 657feec4dc40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:6461, start at 2025-04-04 15:36:45.175683569 +0800 CST m=+4.234188474 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:45.182 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:45.169 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:45.169 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358661201892597 TiDB Cluster ID. tikv_gc_leader_uuid 657feec57340005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:6515, start at 2025-04-04 15:36:45.778841973 +0800 CST m=+4.638395401 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:38:45.786 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:36:45.773 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:26:45.773 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:36:47 CST 2025] <<<<<< START cdc server in api_v2 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.api_v2.87508752.out server --log-file /tmp/tidb_cdc_test/api_v2/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/api_v2/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 18-th check, retry later table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later + set +x + tso='457114178622586885 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114178622586885 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "readallfields"="true" "operationcount"="0" "readproportion"="0" "workload"="core" "dotransactions"="false" "updateproportion"="0" "mysql.port"="4000" "mysql.db"="changefeed_error" "mysql.host"="127.0.0.1" "recordcount"="20" "requestdistribution"="uniform" "insertproportion"="0" "threadcount"="4" "mysql.user"="root" "scanproportion"="0" ********************************************** Run finished, takes 9.750867ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3830.6, Avg(us): 1824, Min(us): 1029, Max(us): 4502, 95th(us): 5000, 99th(us): 5000 [Fri Apr 4 15:36:48 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.53855387.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 19-th check, retry later [Fri Apr 4 15:36:49 CST 2025] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.78277829.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_capture_4.usertable exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:50 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b08588e-856f-4ef1-8bd3-9188920fd873 {"id":"7b08588e-856f-4ef1-8bd3-9188920fd873","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752207} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaf252e2 7b08588e-856f-4ef1-8bd3-9188920fd873 /tidb/cdc/default/default/upstream/7489358659000709430 {"id":7489358659000709430,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b08588e-856f-4ef1-8bd3-9188920fd873 {"id":"7b08588e-856f-4ef1-8bd3-9188920fd873","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752207} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaf252e2 7b08588e-856f-4ef1-8bd3-9188920fd873 /tidb/cdc/default/default/upstream/7489358659000709430 {"id":7489358659000709430,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7b08588e-856f-4ef1-8bd3-9188920fd873 {"id":"7b08588e-856f-4ef1-8bd3-9188920fd873","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752207} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaf252e2 7b08588e-856f-4ef1-8bd3-9188920fd873 /tidb/cdc/default/default/upstream/7489358659000709430 {"id":7489358659000709430,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: cf-blackhole Info: {"upstream_id":7489358659000709430,"namespace":"default","id":"cf-blackhole","sink_uri":"blackhole:","create_time":"2025-04-04T15:36:50.582679833+08:00","start_ts":457114179468525576,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114179468525576,"checkpoint_ts":457114179468525576,"checkpoint_time":"2025-04-04 15:36:50.497"} + endpoints=http://127.0.0.1:2379 + changefeed_id=cf-blackhole + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cf-blackhole -s + info='{ "upstream_id": 7489358659000709430, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 457114179468525576, "checkpoint_time": "2025-04-04 15:36:50.497", "error": null }' + echo '{ "upstream_id": 7489358659000709430, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 457114179468525576, "checkpoint_time": "2025-04-04 15:36:50.497", "error": null }' { "upstream_id": 7489358659000709430, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 457114179468525576, "checkpoint_time": "2025-04-04 15:36:50.497", "error": null } ++ echo '{' '"upstream_id":' 7489358659000709430, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 457114179468525576, '"checkpoint_time":' '"2025-04-04' '15:36:50.497",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358659000709430, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 457114179468525576, '"checkpoint_time":' '"2025-04-04' '15:36:50.497",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + GO111MODULE=on + go run main.go model.go request.go cases.go go: downloading go1.23.4 (linux/amd64) table ddl_manager.finish_mark not exists for 20-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:51 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b {"id":"1c4958db-2403-4641-a04f-ba4b5724450b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96df0 1c4958db-2403-4641-a04f-ba4b5724450b /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b {"id":"1c4958db-2403-4641-a04f-ba4b5724450b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96df0 1c4958db-2403-4641-a04f-ba4b5724450b /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b {"id":"1c4958db-2403-4641-a04f-ba4b5724450b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96df0 1c4958db-2403-4641-a04f-ba4b5724450b /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5438.out cli changefeed create --start-ts=457114178622586885 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error check diff successfully ***************** properties ***************** "readproportion"="0" "mysql.port"="4000" "scanproportion"="0" "threadcount"="2" "readallfields"="true" "mysql.host"="127.0.0.1" "mysql.user"="root" "recordcount"="20" "dotransactions"="false" "updateproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "insertproportion"="0" "mysql.db"="multi_capture_1" "workload"="core" ********************************************** Run finished, takes 10.820495ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2095.6, Avg(us): 1027, Min(us): 417, Max(us): 2108, 95th(us): 2000, 99th(us): 3000 ***************** properties ***************** "scanproportion"="0" "mysql.db"="multi_capture_2" "readproportion"="0" "updateproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "insertproportion"="0" "mysql.host"="127.0.0.1" "workload"="core" "recordcount"="20" "mysql.port"="4000" "dotransactions"="false" "threadcount"="2" "operationcount"="0" "readallfields"="true" ********************************************** Run finished, takes 10.36798ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2171.8, Avg(us): 981, Min(us): 458, Max(us): 1593, 95th(us): 2000, 99th(us): 2000 Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7489358661568166943,"namespace":"default","id":"changefeed-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:36:52.383742913+08:00","start_ts":457114178622586885,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114178622586885,"checkpoint_ts":457114178622586885,"checkpoint_time":"2025-04-04 15:36:47.270"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ***************** properties ***************** "mysql.user"="root" "mysql.port"="4000" "threadcount"="2" "readproportion"="0" "updateproportion"="0" "mysql.db"="multi_capture_3" "readallfields"="true" "workload"="core" "requestdistribution"="uniform" "scanproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "recordcount"="20" "insertproportion"="0" "dotransactions"="false" ********************************************** Run finished, takes 9.960925ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2270.5, Avg(us): 922, Min(us): 458, Max(us): 1663, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "readproportion"="0" "threadcount"="2" "dotransactions"="false" "updateproportion"="0" "scanproportion"="0" "mysql.port"="4000" "workload"="core" "mysql.user"="root" "mysql.host"="127.0.0.1" "insertproportion"="0" "recordcount"="20" "readallfields"="true" "requestdistribution"="uniform" "mysql.db"="multi_capture_4" "operationcount"="0" ********************************************** Run finished, takes 10.947749ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2039.1, Avg(us): 924, Min(us): 474, Max(us): 1654, 95th(us): 2000, 99th(us): 2000 check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:36:52 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f5891d5d-5da4-469d-b247-14984c0aca16 {"id":"f5891d5d-5da4-469d-b247-14984c0aca16","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaecc0dc f5891d5d-5da4-469d-b247-14984c0aca16 /tidb/cdc/default/default/upstream/7489358659426718934 {"id":7489358659426718934,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f5891d5d-5da4-469d-b247-14984c0aca16 {"id":"f5891d5d-5da4-469d-b247-14984c0aca16","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaecc0dc f5891d5d-5da4-469d-b247-14984c0aca16 /tidb/cdc/default/default/upstream/7489358659426718934 {"id":7489358659426718934,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f5891d5d-5da4-469d-b247-14984c0aca16 {"id":"f5891d5d-5da4-469d-b247-14984c0aca16","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752209} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbaecc0dc f5891d5d-5da4-469d-b247-14984c0aca16 /tidb/cdc/default/default/upstream/7489358659426718934 {"id":7489358659426718934,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table processor_stop_delay.t not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 21-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114178622586885, "checkpoint_time": "2025-04-04 15:36:47.270", "error": { "time": "2025-04-04T15:36:52.494472155+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114178622586585 is earlier than or equal to GC safepoint at 457114178622586885" } }' + echo '{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114178622586885, "checkpoint_time": "2025-04-04 15:36:47.270", "error": { "time": "2025-04-04T15:36:52.494472155+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114178622586585 is earlier than or equal to GC safepoint at 457114178622586885" } }' { "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 457114178622586885, "checkpoint_time": "2025-04-04 15:36:47.270", "error": { "time": "2025-04-04T15:36:52.494472155+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114178622586585 is earlier than or equal to GC safepoint at 457114178622586885" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114178622586885, '"checkpoint_time":' '"2025-04-04' '15:36:47.270",' '"error":' '{' '"time":' '"2025-04-04T15:36:52.494472155+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114178622586585 is earlier than or equal to GC safepoint at '457114178622586885"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 457114178622586885, '"checkpoint_time":' '"2025-04-04' '15:36:47.270",' '"error":' '{' '"time":' '"2025-04-04T15:36:52.494472155+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 457114178622586585 is earlier than or equal to GC safepoint at '457114178622586885"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114178622586585 is earlier than or equal to GC safepoint at 457114178622586885' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 457114178622586585 is earlier than or equal to GC safepoint at 457114178622586885 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5495.out cli changefeed resume -c changefeed-error PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table processor_stop_delay.t not exists for 2-th check, retry later check diff successfully Cache restored successfully (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc) 4030750720 bytes in 7.16 secs (562681489 bytes/sec) [Pipeline] { [Pipeline] sh wait process cdc.test exit for 1-th time... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G18 [Pipeline] sh wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 22-th check, retry later Run cases: availability http_proxies sequence GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-smdjj HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=e11d25d6-aad0-497d-886e-fefbf125e11b GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G18 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8 pingcap_tiflow_pull_cdc_integration_test_3827-smdjj PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/availability/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G12 [Pipeline] sh + set +x table changefeed_error.usertable not exists for 1-th check, retry later go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/pingcap/tidb-dashboard v0.0.0-20240326110213-9768844ff5d7 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/sync v0.10.0 go: downloading golang.org/x/time v0.7.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/go-mysql-org/go-mysql v1.12.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-qfqk9 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=0b95bf2c-8ffb-4cc3-b352-a7246c3dc10d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G12 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p pingcap_tiflow_pull_cdc_integration_test_3827-qfqk9 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G11 [Pipeline] sh cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:36:56 CST 2025] <<<<<< run test case multi_capture success! >>>>>> go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 Run cases: resolve_lock move_table autorandom generate_column GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-8bgqv HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=633a4ebd-c9b7-4977-8b6b-e9a08617671a GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G11 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j pingcap_tiflow_pull_cdc_integration_test_3827-8bgqv PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G17 [Pipeline] sh go: downloading github.com/containerd/cgroups v1.0.4 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda Run cases: clustered_index processor_resolved_ts_fallback GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-zpm8c HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=6b752d27-d66c-4551-ab36-c3f9fb38637a GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G17 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6 pingcap_tiflow_pull_cdc_integration_test_3827-zpm8c PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G16 table processor_stop_delay.t exists go: downloading golang.org/x/net v0.33.0 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G14 Run cases: owner_resign processor_etcd_worker_delay sink_hang GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-txjnm HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=b39f3671-4e45-4fa5-852b-76760a0bc17c GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G16 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j pingcap_tiflow_pull_cdc_integration_test_3827-txjnm PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh check diff successfully go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/go-playground/validator/v10 v10.14.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading google.golang.org/protobuf v1.35.2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/aws/smithy-go v1.13.5 Run cases: changefeed_finish force_replicate_table GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-qth0n HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=09055ce6-ec02-4b21-bef3-3d07e052383d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G14 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-qth0n pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G10 [Pipeline] sh go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading filippo.io/edwards25519 v1.1.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 Run cases: default_value simple cdc_server_tips event_filter sql_mode GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 + rm -rf /tmp/tidb_cdc_test POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-n8g2x + mkdir -p /tmp/tidb_cdc_test HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl OLDPWD=/home/jenkins + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G09 RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=bd6411c3-1ae9-45a7-916f-8b25d4e7898f GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G10 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl pingcap_tiflow_pull_cdc_integration_test_3827-n8g2x PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: mysql... <<================= find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/docker/go-units v0.5.0 go: downloading golang.org/x/text v0.21.0 Run cases: gc_safepoint changefeed_pause_resume cli_with_auth savepoint synced_status GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-rkmnv HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=4fbfab8a-d699-445a-af1d-5c497c0214a0 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G09 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1 pingcap_tiflow_pull_cdc_integration_test_3827-rkmnv PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G15 [Pipeline] sh table ddl_manager.finish_mark not exists for 23-th check, retry later go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 table changefeed_error.usertable exists + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G19 Run cases: new_ci_collation batch_add_table multi_rocks ci_collation_compatibility GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-0vf91 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=53aa212c-1a22-46e1-94ef-c8ff67ed69f6 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G15 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-0vf91 pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh check diff failed 1-th time, retry later Run cases: changefeed_fast_fail batch_update_to_no_batch changefeed_resume_with_checkpoint_ts GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-vsll5 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=7585e5e7-8a15-46bd-934d-5b30417a677d GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G19 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-vsll5 pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_fast_fail/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G21 [Pipeline] sh go: downloading github.com/joomcode/errorx v1.0.1 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd Run cases: bank kill_owner_with_ddl owner_remove_table_error GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-m04ht HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=e2d725c2-6e80-4091-97fd-386aa1adb1e8 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G21 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-m04ht pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/bank/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G20 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 Run cases: tidb_mysql_test ddl_reentrant multi_cdc_cluster GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-5fgk4 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=3fe8f245-94c3-43d7-a75e-77ed309c5519 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G20 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-5fgk4 pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/tidb_mysql_test/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G13 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 check diff failed 1-th time, retry later Run cases: tiflash region_merge common_1 GIT_COMMIT=b2a59508c11c86ba701c4af25fc536a9475b82c6 RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect JENKINS_URL=https://do.pingcap.net/jenkins/ EXECUTOR_NUMBER=0 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/pull_cdc_integration_test","buildid":"1908056220913111041","prowjobid":"568410f1-bcd8-4f1c-8c4a-abb24edc023a","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"master","base_sha":"4c631d5951854ac91765b729fe3849d8e8d5ed8a","base_link":"https://github.com/pingcap/tiflow/commit/4c631d5951854ac91765b729fe3849d8e8d5ed8a","pulls":[{"number":12118,"author":"dveeden","sha":"a306f48949e037965a7c1963f500f7e0abbfba51","title":"dm: upgrade go-mysql to v1.12.0","head_ref":"new_gomysql","link":"https://github.com/pingcap/tiflow/pull/12118","commit_link":"https://github.com/pingcap/tiflow/pull/12118/commits/a306f48949e037965a7c1963f500f7e0abbfba51","author_link":"https://github.com/dveeden"}]}} TZ=Asia/Shanghai BUILD_ID=1908056220913111041 POD_LABEL=pingcap_tiflow_pull_cdc_integration_test_3827-zxhg8 HOSTNAME=pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6 OLDPWD=/home/jenkins RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=changes JENKINS_NODE_COOKIE=96741198-533b-478d-9297-a154af9832e8 GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct JAVA_HOME=/usr/lib/jvm/jre-openjdk JOB_BASE_NAME=pull_cdc_integration_test WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test@tmp KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true GIT_URL=https://github.com/PingCAP-QE/ci.git CLASSPATH= KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 FILE_SERVER_URL=http://fileserver.pingcap.net POD_CONTAINER=golang GOPATH=/go BUILD_NUMBER=3827 USE_BAZEL_VERSION=6.5.0 KUBERNETES_PORT=tcp://10.233.0.1:443 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test PWD=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ HOME=/home/jenkins NODE_NAME=pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 JENKINS_HOME=/var/jenkins_home JOB_NAME=pingcap/tiflow/pull_cdc_integration_test RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=tests KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_PORT_443_TCP_PORT=443 GIT_PREVIOUS_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 HUDSON_HOME=/var/jenkins_home JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/display/redirect PROW_JOB_ID=568410f1-bcd8-4f1c-8c4a-abb24edc023a TEST_GROUP=G13 TICDC_CODECOV_TOKEN=**** KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/display/redirect?page=artifacts BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/3827/ TERM=xterm STAGE_NAME=Test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/pull_cdc_integration_test/ BUILD_DISPLAY_NAME=#3827 SHLVL=5 GIT_BRANCH=origin/main BUILD_TAG=jenkins-pingcap-tiflow-pull_cdc_integration_test-3827 KUBERNETES_SERVICE_PORT=443 NODE_LABELS=pingcap_tiflow_pull_cdc_integration_test_3827-zxhg8 pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6 PATH=/go/bin:/root/.cargo/bin:/usr/local/go/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/gradle-7.4.2/bin:/opt/apache-maven-3.8.8/bin:/usr/local/pulsar/bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/../../scripts/bin TICDC_COVERALLS_TOKEN=**** GIT_PREVIOUS_SUCCESSFUL_COMMIT=3e31ec89ba643faa364a78dbc7f062bb55f9c486 KUBERNETES_SERVICE_HOST=10.233.0.1 JENKINS_SERVER_COOKIE=durable-abe98160c26b91d7ad4012fc504dcc1b24097e77dc15bbc9f9f12631da13fdbf _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible table ddl_manager.finish_mark not exists for 24-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/clustered_index Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 The 1 times to try to start tidb cluster... check diff successfully ***************** properties ***************** "recordcount"="20" "mysql.user"="root" "requestdistribution"="uniform" "readallfields"="true" "scanproportion"="0" "readproportion"="0" "workload"="core" "updateproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_error" "mysql.port"="4000" "dotransactions"="false" "insertproportion"="0" "threadcount"="4" "operationcount"="0" ********************************************** Run finished, takes 5.267211ms INSERT - Takes(s): 0.0, Count: 20, OPS: 5970.6, Avg(us): 863, Min(us): 466, Max(us): 2083, 95th(us): 3000, 99th(us): 3000 check diff successfully {"id":"1c4958db-2403-4641-a04f-ba4b5724450b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752209} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b + [[ /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 check diff failed 2-th time, retry later go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading go.uber.org/mock v0.4.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/jfcg/sorty/v2 v2.1.0 start tidb cluster in /tmp/tidb_cdc_test/availability Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/jfcg/sixb v1.3.8 go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/apache/arrow/go/v12 v12.0.1 start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 25-th check, retry later go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/ardielle/ardielle-go v1.5.2 start tidb cluster in /tmp/tidb_cdc_test/new_ci_collation Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/changefeed_fast_fail Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b + [[ /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/klauspost/cpuid/v2 v2.2.4 start tidb cluster in /tmp/tidb_cdc_test/bank Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... go: downloading k8s.io/apimachinery v0.29.11 The 1 times to try to start tidb cluster... go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 26-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 27-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... start tidb cluster in /tmp/tidb_cdc_test/many_pk_or_uk Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/tidb_mysql_test Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b + [[ /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/1c4958db-2403-4641-a04f-ba4b5724450b + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 28-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status_with_redo + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358765827600768 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbcc40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4, pid:1346, start at 2025-04-04 15:37:08.695118258 +0800 CST m=+2.017048617 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.702 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.707 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.707 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 29-th check, retry later wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358766531781670 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb3100008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:1278, start at 2025-04-04 15:37:08.049520019 +0800 CST m=+2.156903206 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.057 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.061 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.061 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358766531781670 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb3100008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:1278, start at 2025-04-04 15:37:08.049520019 +0800 CST m=+2.156903206 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.057 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.061 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.061 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358771232785640 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb310000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:1360, start at 2025-04-04 15:37:08.059328609 +0800 CST m=+2.073398404 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.067 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.062 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.062 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiDB... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358772098331220 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb440000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:1238, start at 2025-04-04 15:37:08.151284956 +0800 CST m=+1.998434621 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.158 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.162 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.162 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358772098331220 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb440000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:1238, start at 2025-04-04 15:37:08.151284956 +0800 CST m=+1.998434621 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.158 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.162 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.162 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358770756644672 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb818000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:1306, start at 2025-04-04 15:37:08.390314507 +0800 CST m=+2.143766611 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.397 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.358 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.358 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bank/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bank/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/bank/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bank/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/bank/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358766378819224 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb194000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:1388, start at 2025-04-04 15:37:07.955784289 +0800 CST m=+2.177386041 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:07.963 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:07.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:07.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358766378819224 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb194000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:1388, start at 2025-04-04 15:37:07.955784289 +0800 CST m=+2.177386041 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:07.963 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:07.941 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:07.941 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358762478544814 TiDB Cluster ID. tikv_gc_leader_uuid 657feedb3ec0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:1445, start at 2025-04-04 15:37:08.138645416 +0800 CST m=+2.286750818 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.146 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.141 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.141 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/availability/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/availability/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/availability/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/availability/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/availability/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358774378410733 TiDB Cluster ID. tikv_gc_leader_uuid 657feedc080000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:1212, start at 2025-04-04 15:37:08.940811218 +0800 CST m=+2.276005178 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.949 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.946 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.946 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358774378410733 TiDB Cluster ID. tikv_gc_leader_uuid 657feedc080000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:1212, start at 2025-04-04 15:37:08.940811218 +0800 CST m=+2.276005178 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.949 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.946 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.946 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358763232890287 TiDB Cluster ID. tikv_gc_leader_uuid 657feedc2000007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:1291, start at 2025-04-04 15:37:09.02840664 +0800 CST m=+2.279988384 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:09.037 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:09.042 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:09.042 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358770960706378 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbd00000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6, pid:1280, start at 2025-04-04 15:37:08.694511367 +0800 CST m=+2.252142181 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.703 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.672 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.672 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358770960706378 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbd00000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6, pid:1280, start at 2025-04-04 15:37:08.694511367 +0800 CST m=+2.252142181 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.703 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.672 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.672 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358766496932560 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbea0000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6, pid:1351, start at 2025-04-04 15:37:08.792599223 +0800 CST m=+2.275459974 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.800 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.776 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.776 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/clustered_index/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/clustered_index/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358773118019169 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbce4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:1348, start at 2025-04-04 15:37:08.683461662 +0800 CST m=+2.348135477 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.692 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.665 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.665 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358773118019169 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbce4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:1348, start at 2025-04-04 15:37:08.683461662 +0800 CST m=+2.348135477 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.692 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.665 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.665 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358771558066838 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbce4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:1423, start at 2025-04-04 15:37:08.698209524 +0800 CST m=+2.283402221 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.706 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.703 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.703 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358765827600768 TiDB Cluster ID. tikv_gc_leader_uuid 657feedbcc40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4, pid:1346, start at 2025-04-04 15:37:08.695118258 +0800 CST m=+2.017048617 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.702 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.707 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.707 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358770473041771 TiDB Cluster ID. tikv_gc_leader_uuid 657feedc0b40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4, pid:1423, start at 2025-04-04 15:37:08.911988929 +0800 CST m=+2.157785446 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:08.918 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:08.909 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:08.909 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:11 CST 2025] <<<<<< run test case processor_stop_delay success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 30-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358778135468459 TiDB Cluster ID. tikv_gc_leader_uuid 657feedd1f8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:1272, start at 2025-04-04 15:37:10.049243835 +0800 CST m=+2.414468612 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:10.057 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:10.064 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:10.064 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358778135468459 TiDB Cluster ID. tikv_gc_leader_uuid 657feedd1f8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:1272, start at 2025-04-04 15:37:10.049243835 +0800 CST m=+2.414468612 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:10.057 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:10.064 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:10.064 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358779948375212 TiDB Cluster ID. tikv_gc_leader_uuid 657feedd3b40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:1345, start at 2025-04-04 15:37:10.133989667 +0800 CST m=+2.390407968 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:10.142 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:10.125 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:10.125 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2688.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358777413183495 TiDB Cluster ID. tikv_gc_leader_uuid 657feedd2500017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:1211, start at 2025-04-04 15:37:10.07583138 +0800 CST m=+2.270127076 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:10.084 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:10.081 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:10.081 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358777413183495 TiDB Cluster ID. tikv_gc_leader_uuid 657feedd2500017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:1211, start at 2025-04-04 15:37:10.07583138 +0800 CST m=+2.270127076 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:10.084 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:10.081 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:10.081 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358770853600326 TiDB Cluster ID. tikv_gc_leader_uuid 657feedd30c0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:1286, start at 2025-04-04 15:37:10.086878227 +0800 CST m=+2.202258760 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:10.094 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:10.136 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:10.136 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358780560304485 TiDB Cluster ID. tikv_gc_leader_uuid 657feedef90000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:1342, start at 2025-04-04 15:37:11.925347718 +0800 CST m=+3.695410595 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:11.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:11.908 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:11.908 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358780560304485 TiDB Cluster ID. tikv_gc_leader_uuid 657feedef90000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:1342, start at 2025-04-04 15:37:11.925347718 +0800 CST m=+3.695410595 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:11.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:11.908 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:11.908 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358777415102137 TiDB Cluster ID. tikv_gc_leader_uuid 657feedef84000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:1412, start at 2025-04-04 15:37:11.920744062 +0800 CST m=+3.569672054 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:11.929 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:11.905 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:11.905 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:37:12 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.26422644.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:37:12 CST 2025] <<<<<< START cdc server in bank case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.26932695.out server --log-file /tmp/tidb_cdc_test/bank/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bank/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:37:12 CST 2025] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.26772679.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358782451691000 TiDB Cluster ID. tikv_gc_leader_uuid 657feede624000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:1285, start at 2025-04-04 15:37:11.352537653 +0800 CST m=+2.621959360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:11.361 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:11.355 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:11.355 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358782451691000 TiDB Cluster ID. tikv_gc_leader_uuid 657feede624000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:1285, start at 2025-04-04 15:37:11.352537653 +0800 CST m=+2.621959360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:11.361 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:11.355 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:11.355 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358778222953368 TiDB Cluster ID. tikv_gc_leader_uuid 657feede6f4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:1351, start at 2025-04-04 15:37:11.391900457 +0800 CST m=+2.559762352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:11.440 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:11.408 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:11.408 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 4 15:37:12 CST 2025] <<<<<< START cdc server in clustered_index case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.clustered_index.26712673.out server --log-file /tmp/tidb_cdc_test/clustered_index/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/clustered_index/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:37:12 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.57475749.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 4 15:37:13 CST 2025] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.26702672.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2665.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 31-th check, retry later + set +x + tso='457114185234120705 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114185234120705 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2679.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2714.out cli tso query --pd=http://127.0.0.1:2379 [Fri Apr 4 15:37:14 CST 2025] <<<<<< START cdc server in new_ci_collation case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.new_ci_collation.26722674.out server --log-file /tmp/tidb_cdc_test/new_ci_collation/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/new_ci_collation/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358790259337600 TiDB Cluster ID. tikv_gc_leader_uuid 657feedf918000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:1303, start at 2025-04-04 15:37:12.556388151 +0800 CST m=+2.191622127 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:12.564 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:12.567 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:12.567 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358790259337600 TiDB Cluster ID. tikv_gc_leader_uuid 657feedf918000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:1303, start at 2025-04-04 15:37:12.556388151 +0800 CST m=+2.191622127 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:12.564 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:12.567 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:12.567 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358789271953599 TiDB Cluster ID. tikv_gc_leader_uuid 657feedfcf40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:1380, start at 2025-04-04 15:37:12.776397543 +0800 CST m=+2.326940352 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:12.784 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:12.765 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:12.765 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='457114185471361025 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114185471361025 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:37:14 CST 2025] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.26972699.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2641.out cli tso query --pd=http://127.0.0.1:2379 run test case test_kill_owner [Fri Apr 4 15:37:15 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.27442746.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 32-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:15 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3295 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/388f4901-9d3c-4c4d-a597-bb2a9eedb6e5 {"id":"388f4901-9d3c-4c4d-a597-bb2a9eedb6e5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96eeb 388f4901-9d3c-4c4d-a597-bb2a9eedb6e5 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489358661568166943,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:36:52.383742913+08:00","start-ts":457114178622586885,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:37:13.365288728+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114185451438085} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114181860065281,"min-table-barrier-ts":457114181860065281,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/388f4901-9d3c-4c4d-a597-bb2a9eedb6e5 {"id":"388f4901-9d3c-4c4d-a597-bb2a9eedb6e5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96eeb 388f4901-9d3c-4c4d-a597-bb2a9eedb6e5 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489358661568166943,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:36:52.383742913+08:00","start-ts":457114178622586885,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:37:13.365288728+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114185451438085} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114181860065281,"min-table-barrier-ts":457114181860065281,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/388f4901-9d3c-4c4d-a597-bb2a9eedb6e5 {"id":"388f4901-9d3c-4c4d-a597-bb2a9eedb6e5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96eeb 388f4901-9d3c-4c4d-a597-bb2a9eedb6e5 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7489358661568166943,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:36:52.383742913+08:00","start-ts":457114178622586885,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"pending","error":{"time":"2025-04-04T15:37:13.365288728+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114185451438085} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":457114181860065281,"min-table-barrier-ts":457114181860065281,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114181860065281, "checkpoint_time": "2025-04-04 15:36:59.620", "error": { "time": "2025-04-04T15:37:13.365288728+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114181860065281, "checkpoint_time": "2025-04-04 15:36:59.620", "error": { "time": "2025-04-04T15:37:13.365288728+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 457114181860065281, "checkpoint_time": "2025-04-04 15:36:59.620", "error": { "time": "2025-04-04T15:37:13.365288728+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114181860065281, '"checkpoint_time":' '"2025-04-04' '15:36:59.620",' '"error":' '{' '"time":' '"2025-04-04T15:37:13.365288728+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114181860065281, '"checkpoint_time":' '"2025-04-04' '15:36:59.620",' '"error":' '{' '"time":' '"2025-04-04T15:37:13.365288728+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5843.out cli changefeed remove -c changefeed-error + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358790029552117 TiDB Cluster ID. tikv_gc_leader_uuid 657feee0d980013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:1294, start at 2025-04-04 15:37:13.866780001 +0800 CST m=+2.285226776 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:13.875 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:13.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:13.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358790029552117 TiDB Cluster ID. tikv_gc_leader_uuid 657feee0d980013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:1294, start at 2025-04-04 15:37:13.866780001 +0800 CST m=+2.285226776 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:13.875 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:13.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:13.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358791763825066 TiDB Cluster ID. tikv_gc_leader_uuid 657feee0d98000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:1372, start at 2025-04-04 15:37:13.863689359 +0800 CST m=+2.203722004 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:13.874 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:13.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:13.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:15 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4dd99cde-0247-465e-870a-1a549d30416a {"id":"4dd99cde-0247-465e-870a-1a549d30416a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752232} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb562fe0 4dd99cde-0247-465e-870a-1a549d30416a /tidb/cdc/default/default/upstream/7489358766531781670 {"id":7489358766531781670,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4dd99cde-0247-465e-870a-1a549d30416a {"id":"4dd99cde-0247-465e-870a-1a549d30416a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752232} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb562fe0 4dd99cde-0247-465e-870a-1a549d30416a /tidb/cdc/default/default/upstream/7489358766531781670 {"id":7489358766531781670,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4dd99cde-0247-465e-870a-1a549d30416a {"id":"4dd99cde-0247-465e-870a-1a549d30416a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752232} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb562fe0 4dd99cde-0247-465e-870a-1a549d30416a /tidb/cdc/default/default/upstream/7489358766531781670 {"id":7489358766531781670,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.cli.2749.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ Create changefeed successfully! ID: 530b5932-fe58-4bec-b163-c2b1773673ff Info: {"upstream_id":7489358766531781670,"namespace":"default","id":"530b5932-fe58-4bec-b163-c2b1773673ff","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:37:15.915898314+08:00","start_ts":457114186117545985,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114186117545985,"checkpoint_ts":457114186117545985,"checkpoint_time":"2025-04-04 15:37:15.861"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:15 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc11d138-27ca-4b40-a40a-2f803fd9a1fb {"id":"dc11d138-27ca-4b40-a40a-2f803fd9a1fb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501bd3 dc11d138-27ca-4b40-a40a-2f803fd9a1fb /tidb/cdc/default/default/upstream/7489358773118019169 {"id":7489358773118019169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc11d138-27ca-4b40-a40a-2f803fd9a1fb {"id":"dc11d138-27ca-4b40-a40a-2f803fd9a1fb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501bd3 dc11d138-27ca-4b40-a40a-2f803fd9a1fb /tidb/cdc/default/default/upstream/7489358773118019169 {"id":7489358773118019169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc11d138-27ca-4b40-a40a-2f803fd9a1fb {"id":"dc11d138-27ca-4b40-a40a-2f803fd9a1fb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501bd3 dc11d138-27ca-4b40-a40a-2f803fd9a1fb /tidb/cdc/default/default/upstream/7489358773118019169 {"id":7489358773118019169,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2741.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 + set +x + tso='457114185755787265 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114185755787265 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:37:16 CST 2025] <<<<<< START cdc server in changefeed_fast_fail case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.27182720.out server --log-file /tmp/tidb_cdc_test/changefeed_fast_fail/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_fast_fail/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:16 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34b366f8-bd0e-45c0-992a-e89bfb95bbd8 {"id":"34b366f8-bd0e-45c0-992a-e89bfb95bbd8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4e9bd7 34b366f8-bd0e-45c0-992a-e89bfb95bbd8 /tidb/cdc/default/default/upstream/7489358770960706378 {"id":7489358770960706378,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34b366f8-bd0e-45c0-992a-e89bfb95bbd8 {"id":"34b366f8-bd0e-45c0-992a-e89bfb95bbd8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4e9bd7 34b366f8-bd0e-45c0-992a-e89bfb95bbd8 /tidb/cdc/default/default/upstream/7489358770960706378 {"id":7489358770960706378,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34b366f8-bd0e-45c0-992a-e89bfb95bbd8 {"id":"34b366f8-bd0e-45c0-992a-e89bfb95bbd8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4e9bd7 34b366f8-bd0e-45c0-992a-e89bfb95bbd8 /tidb/cdc/default/default/upstream/7489358770960706378 {"id":7489358770960706378,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: a540868c-d891-4a61-bc64-c0d7f4d67c00 Info: {"upstream_id":7489358770960706378,"namespace":"default","id":"a540868c-d891-4a61-bc64-c0d7f4d67c00","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:16.188521906+08:00","start_ts":457114185347366913,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114185347366913,"checkpoint_ts":457114185347366913,"checkpoint_time":"2025-04-04 15:37:12.923"} + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:15 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47242da2-0537-4aae-ac83-d5bf807520be {"id":"47242da2-0537-4aae-ac83-d5bf807520be","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752232} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5340d8 47242da2-0537-4aae-ac83-d5bf807520be /tidb/cdc/default/default/upstream/7489358772098331220 {"id":7489358772098331220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47242da2-0537-4aae-ac83-d5bf807520be {"id":"47242da2-0537-4aae-ac83-d5bf807520be","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752232} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5340d8 47242da2-0537-4aae-ac83-d5bf807520be /tidb/cdc/default/default/upstream/7489358772098331220 {"id":7489358772098331220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47242da2-0537-4aae-ac83-d5bf807520be {"id":"47242da2-0537-4aae-ac83-d5bf807520be","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752232} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5340d8 47242da2-0537-4aae-ac83-d5bf807520be /tidb/cdc/default/default/upstream/7489358772098331220 {"id":7489358772098331220,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x 0 < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:16 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c663f4ee-8cb3-4e0a-95de-a46eebe562fb {"id":"c663f4ee-8cb3-4e0a-95de-a46eebe562fb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5149d9 c663f4ee-8cb3-4e0a-95de-a46eebe562fb /tidb/cdc/default/default/upstream/7489358765827600768 {"id":7489358765827600768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c663f4ee-8cb3-4e0a-95de-a46eebe562fb {"id":"c663f4ee-8cb3-4e0a-95de-a46eebe562fb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5149d9 c663f4ee-8cb3-4e0a-95de-a46eebe562fb /tidb/cdc/default/default/upstream/7489358765827600768 {"id":7489358765827600768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c663f4ee-8cb3-4e0a-95de-a46eebe562fb {"id":"c663f4ee-8cb3-4e0a-95de-a46eebe562fb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5149d9 c663f4ee-8cb3-4e0a-95de-a46eebe562fb /tidb/cdc/default/default/upstream/7489358765827600768 {"id":7489358765827600768,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: b07e96df-5865-4319-8f11-66d85e2da821 Info: {"upstream_id":7489358773118019169,"namespace":"default","id":"b07e96df-5865-4319-8f11-66d85e2da821","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:16.415023756+08:00","start_ts":457114186234200066,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114186234200066,"checkpoint_ts":457114186234200066,"checkpoint_time":"2025-04-04 15:37:16.306"} PASS + set +x + tso='457114185907044353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114185907044353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:37:16 CST 2025] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.27402742.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 coverage: 2.4% of statements in github.com/pingcap/tiflow/... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.cli.2769.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='457114186024222721 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114186024222721 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:37:16 CST 2025] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.26782680.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:17 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43572bfb-3b3a-4210-a183-fb03c9d382ea {"id":"43572bfb-3b3a-4210-a183-fb03c9d382ea","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752234} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5538d6 43572bfb-3b3a-4210-a183-fb03c9d382ea /tidb/cdc/default/default/upstream/7489358778135468459 {"id":7489358778135468459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43572bfb-3b3a-4210-a183-fb03c9d382ea {"id":"43572bfb-3b3a-4210-a183-fb03c9d382ea","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752234} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5538d6 43572bfb-3b3a-4210-a183-fb03c9d382ea /tidb/cdc/default/default/upstream/7489358778135468459 {"id":7489358778135468459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43572bfb-3b3a-4210-a183-fb03c9d382ea {"id":"43572bfb-3b3a-4210-a183-fb03c9d382ea","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752234} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5538d6 43572bfb-3b3a-4210-a183-fb03c9d382ea /tidb/cdc/default/default/upstream/7489358778135468459 {"id":7489358778135468459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Changefeed remove successfully. ID: changefeed-error CheckpointTs: 457114181860065281 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x [Fri Apr 4 15:37:17 CST 2025] <<<<<< START cdc server in bank case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.27832785.out server --log-file /tmp/tidb_cdc_test/bank/cdcdown.log --log-level debug --data-dir /tmp/tidb_cdc_test/bank/cdc_datadown --cluster-id default --addr 127.0.0.1:8400 --pd http://127.0.0.1:2479 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8400 failed: Connection refused * Failed to connect to 127.0.0.1 port 8400: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 08e0ac31-d440-44c8-9b65-33d2b61b6de8 Info: {"upstream_id":7489358778135468459,"namespace":"default","id":"08e0ac31-d440-44c8-9b65-33d2b61b6de8","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:37:17.406738704+08:00","start_ts":457114185659580419,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114185659580419,"checkpoint_ts":457114185659580419,"checkpoint_time":"2025-04-04 15:37:14.114"} + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2733.out cli tso query --pd=http://127.0.0.1:2379 + set +x cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' run task failed 1-th time, retry later table new_ci_collation_test.t1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 33-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:17 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9dc42d75-7cad-406e-8da0-5d61e8f09404 {"id":"9dc42d75-7cad-406e-8da0-5d61e8f09404","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb516ff2 9dc42d75-7cad-406e-8da0-5d61e8f09404 /tidb/cdc/default/default/upstream/7489358774378410733 {"id":7489358774378410733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9dc42d75-7cad-406e-8da0-5d61e8f09404 {"id":"9dc42d75-7cad-406e-8da0-5d61e8f09404","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb516ff2 9dc42d75-7cad-406e-8da0-5d61e8f09404 /tidb/cdc/default/default/upstream/7489358774378410733 {"id":7489358774378410733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9dc42d75-7cad-406e-8da0-5d61e8f09404 {"id":"9dc42d75-7cad-406e-8da0-5d61e8f09404","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb516ff2 9dc42d75-7cad-406e-8da0-5d61e8f09404 /tidb/cdc/default/default/upstream/7489358774378410733 {"id":7489358774378410733,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2755.out cli changefeed create --start-ts=457114185471361025 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 48bbc3f1-4b9e-4eed-b01e-1ebfc83dfa06 Info: {"upstream_id":7489358774378410733,"namespace":"default","id":"48bbc3f1-4b9e-4eed-b01e-1ebfc83dfa06","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:18.356123171+08:00","start_ts":457114185471361025,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114185471361025,"checkpoint_ts":457114185471361025,"checkpoint_time":"2025-04-04 15:37:13.396"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114186433953793 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114186433953793 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:37:18 CST 2025] <<<<<< START cdc server in tidb_mysql_test case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.27952797.out server --log-file /tmp/tidb_cdc_test/tidb_mysql_test/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tidb_mysql_test/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:18 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e213e98-2124-4b4c-8d33-4c788b85d3d1 {"id":"2e213e98-2124-4b4c-8d33-4c788b85d3d1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb09 2e213e98-2124-4b4c-8d33-4c788b85d3d1 /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e213e98-2124-4b4c-8d33-4c788b85d3d1 {"id":"2e213e98-2124-4b4c-8d33-4c788b85d3d1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb09 2e213e98-2124-4b4c-8d33-4c788b85d3d1 /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e213e98-2124-4b4c-8d33-4c788b85d3d1 {"id":"2e213e98-2124-4b4c-8d33-4c788b85d3d1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb09 2e213e98-2124-4b4c-8d33-4c788b85d3d1 /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2805.out cli changefeed create --start-ts=457114185234120705 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully Create changefeed successfully! ID: cb28ccbc-03e8-4199-beb7-a5edfeec8da7 Info: {"upstream_id":7489358766378819224,"namespace":"default","id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:19.080158588+08:00","start_ts":457114185234120705,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114185234120705,"checkpoint_ts":457114185234120705,"checkpoint_time":"2025-04-04 15:37:12.491"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:19 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e53afec-bc5e-4b9a-b64d-0acb63fa4c27 {"id":"9e53afec-bc5e-4b9a-b64d-0acb63fa4c27","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752236} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb55deea 9e53afec-bc5e-4b9a-b64d-0acb63fa4c27 /tidb/cdc/default/default/upstream/7489358777413183495 {"id":7489358777413183495,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e53afec-bc5e-4b9a-b64d-0acb63fa4c27 {"id":"9e53afec-bc5e-4b9a-b64d-0acb63fa4c27","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752236} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb55deea 9e53afec-bc5e-4b9a-b64d-0acb63fa4c27 /tidb/cdc/default/default/upstream/7489358777413183495 {"id":7489358777413183495,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e53afec-bc5e-4b9a-b64d-0acb63fa4c27 {"id":"9e53afec-bc5e-4b9a-b64d-0acb63fa4c27","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752236} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb55deea 9e53afec-bc5e-4b9a-b64d-0acb63fa4c27 /tidb/cdc/default/default/upstream/7489358777413183495 {"id":7489358777413183495,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2782.out cli changefeed create --start-ts=457114185755787265 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-fast-fail table clustered_index_test.t0 exists table clustered_index_test.t1 not exists for 1-th check, retry later Create changefeed successfully! ID: changefeed-fast-fail Info: {"upstream_id":7489358777413183495,"namespace":"default","id":"changefeed-fast-fail","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:37:19.479016299+08:00","start_ts":457114185755787265,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114185755787265,"checkpoint_ts":457114185755787265,"checkpoint_time":"2025-04-04 15:37:14.481"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:19 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bdd713a4-6050-4dff-8ff8-6b7da5f51e88 {"id":"bdd713a4-6050-4dff-8ff8-6b7da5f51e88","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752236} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5b70e1 bdd713a4-6050-4dff-8ff8-6b7da5f51e88 /tidb/cdc/default/default/upstream/7489358780560304485 {"id":7489358780560304485,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bdd713a4-6050-4dff-8ff8-6b7da5f51e88 {"id":"bdd713a4-6050-4dff-8ff8-6b7da5f51e88","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752236} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5b70e1 bdd713a4-6050-4dff-8ff8-6b7da5f51e88 /tidb/cdc/default/default/upstream/7489358780560304485 {"id":7489358780560304485,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bdd713a4-6050-4dff-8ff8-6b7da5f51e88 {"id":"bdd713a4-6050-4dff-8ff8-6b7da5f51e88","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752236} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5b70e1 bdd713a4-6050-4dff-8ff8-6b7da5f51e88 /tidb/cdc/default/default/upstream/7489358780560304485 {"id":7489358780560304485,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2794.out cli changefeed create --start-ts=457114185907044353 --sink-uri=mysql://normal:123456@127.0.0.1:3306/tidb-txn-mode=pessimistic wait process cdc.test exit for 1-th time... + set +x + tso='457114186762682369 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114186762682369 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:37:19 CST 2025] <<<<<< START cdc server in many_pk_or_uk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.27722774.out server --log-file /tmp/tidb_cdc_test/many_pk_or_uk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/many_pk_or_uk/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x succeed to verify meta placement rules table new_ci_collation_test.t1 exists table new_ci_collation_test.t2 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 34-th check, retry later Create changefeed successfully! ID: ff0e33b2-6330-4071-bafd-37ae71843fd5 Info: {"upstream_id":7489358780560304485,"namespace":"default","id":"ff0e33b2-6330-4071-bafd-37ae71843fd5","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/tidb-txn-mode=pessimistic","create_time":"2025-04-04T15:37:20.002464741+08:00","start_ts":457114185907044353,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114185907044353,"checkpoint_ts":457114185907044353,"checkpoint_time":"2025-04-04 15:37:15.058"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:20 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ba05a9e5-ae32-4966-a0a3-fb28ac105111 {"id":"ba05a9e5-ae32-4966-a0a3-fb28ac105111","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752237} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5d62e2 ba05a9e5-ae32-4966-a0a3-fb28ac105111 /tidb/cdc/default/default/upstream/7489358782451691000 {"id":7489358782451691000,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ba05a9e5-ae32-4966-a0a3-fb28ac105111 {"id":"ba05a9e5-ae32-4966-a0a3-fb28ac105111","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752237} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5d62e2 ba05a9e5-ae32-4966-a0a3-fb28ac105111 /tidb/cdc/default/default/upstream/7489358782451691000 {"id":7489358782451691000,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ba05a9e5-ae32-4966-a0a3-fb28ac105111 {"id":"ba05a9e5-ae32-4966-a0a3-fb28ac105111","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752237} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb5d62e2 ba05a9e5-ae32-4966-a0a3-fb28ac105111 /tidb/cdc/default/default/upstream/7489358782451691000 {"id":7489358782451691000,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: bd9fc696-8826-4824-a9a6-795b346c73d0 Info: {"upstream_id":7489358782451691000,"namespace":"default","id":"bd9fc696-8826-4824-a9a6-795b346c73d0","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:20.26676644+08:00","start_ts":457114186024222721,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114186024222721,"checkpoint_ts":457114186024222721,"checkpoint_time":"2025-04-04 15:37:15.505"} wait process cdc.test exit for 2-th time... cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table owner_resign.t1 exists check diff failed 1-th time, retry later table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8400 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8400 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:20 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/390890fb-ce62-4dcb-a8a1-662c1a2ded51 {"id":"390890fb-ce62-4dcb-a8a1-662c1a2ded51","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752237} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffbb56620c 390890fb-ce62-4dcb-a8a1-662c1a2ded51 /tidb/cdc/default/default/upstream/7489358771232785640 {"id":7489358771232785640,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/390890fb-ce62-4dcb-a8a1-662c1a2ded51 {"id":"390890fb-ce62-4dcb-a8a1-662c1a2ded51","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752237} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffbb56620c 390890fb-ce62-4dcb-a8a1-662c1a2ded51 /tidb/cdc/default/default/upstream/7489358771232785640 {"id":7489358771232785640,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/390890fb-ce62-4dcb-a8a1-662c1a2ded51 {"id":"390890fb-ce62-4dcb-a8a1-662c1a2ded51","address":"127.0.0.1:8400","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752237} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/351195ffbb56620c 390890fb-ce62-4dcb-a8a1-662c1a2ded51 /tidb/cdc/default/default/upstream/7489358771232785640 {"id":7489358771232785640,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.cli.2845.out cli changefeed create --sink-uri=blackhole:// -c changefeed-for-find-finished-ts --server http://127.0.0.1:8400 Create changefeed successfully! ID: changefeed-for-find-finished-ts Info: {"upstream_id":7489358771232785640,"namespace":"default","id":"changefeed-for-find-finished-ts","sink_uri":"blackhole:","create_time":"2025-04-04T15:37:20.772683139+08:00","start_ts":457114187389206531,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114187389206531,"checkpoint_ts":457114187389206531,"checkpoint_time":"2025-04-04 15:37:20.712"} PASS VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358816165049312 TiDB Cluster ID. tikv_gc_leader_uuid 657feee5da40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:9910, start at 2025-04-04 15:37:18.961070147 +0800 CST m=+2.221042710 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:18.969 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:18.953 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:18.953 +0800 All versions after safe point can be accessed. (DO NOT EDIT) coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-fast-fail failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-fast-fail + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-fast-fail -s VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358816165049312 TiDB Cluster ID. tikv_gc_leader_uuid 657feee5da40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:9910, start at 2025-04-04 15:37:18.961070147 +0800 CST m=+2.221042710 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:18.969 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:18.953 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:18.953 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358815732970291 TiDB Cluster ID. tikv_gc_leader_uuid 657feee5f30000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:9989, start at 2025-04-04 15:37:19.070286354 +0800 CST m=+2.226262805 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:19.081 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:19.123 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:19.123 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... + info='{ "upstream_id": 7489358777413183495, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 457114185755787265, "checkpoint_time": "2025-04-04 15:37:14.481", "error": { "time": "2025-04-04T15:37:19.560678864+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7489358777413183495, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 457114185755787265, "checkpoint_time": "2025-04-04 15:37:14.481", "error": { "time": "2025-04-04T15:37:19.560678864+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7489358777413183495, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 457114185755787265, "checkpoint_time": "2025-04-04 15:37:14.481", "error": { "time": "2025-04-04T15:37:19.560678864+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ echo '{' '"upstream_id":' 7489358777413183495, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 457114185755787265, '"checkpoint_time":' '"2025-04-04' '15:37:14.481",' '"error":' '{' '"time":' '"2025-04-04T15:37:19.560678864+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489358777413183495, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 457114185755787265, '"checkpoint_time":' '"2025-04-04' '15:37:14.481",' '"error":' '{' '"time":' '"2025-04-04T15:37:19.560678864+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully check diff successfully wait process cdc.test exit for 4-th time... table clustered_index_test.t1 exists table clustered_index_test.t2 not exists for 1-th check, retry later + set +x go: downloading go1.23.4 (linux/amd64) check diff successfully check_safepoint_forward http://127.0.0.1:2379 7489358772098331220 457114187245027331 457114186078486533 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:21 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b0d801b5-d49f-4c1a-b438-2e547557a96f {"id":"b0d801b5-d49f-4c1a-b438-2e547557a96f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752238} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb65a1cf b0d801b5-d49f-4c1a-b438-2e547557a96f /tidb/cdc/default/default/upstream/7489358790259337600 {"id":7489358790259337600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b0d801b5-d49f-4c1a-b438-2e547557a96f {"id":"b0d801b5-d49f-4c1a-b438-2e547557a96f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752238} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb65a1cf b0d801b5-d49f-4c1a-b438-2e547557a96f /tidb/cdc/default/default/upstream/7489358790259337600 {"id":7489358790259337600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b0d801b5-d49f-4c1a-b438-2e547557a96f {"id":"b0d801b5-d49f-4c1a-b438-2e547557a96f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752238} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb65a1cf b0d801b5-d49f-4c1a-b438-2e547557a96f /tidb/cdc/default/default/upstream/7489358790259337600 {"id":7489358790259337600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 142 0 0 100 142 0 1479 --:--:-- --:--:-- --:--:-- 1479 mysql_test start go: downloading github.com/maxshuang/mysql-tester v0.0.2 wait process cdc.test exit for 5-th time... table ddl_manager.finish_mark not exists for 35-th check, retry later "is-owner": true, run task successfully wait process cdc.test exit for 1-th time... + set +x + GO111MODULE=on + go run bank.go case.go -u 'root@tcp(127.0.0.1:4000)/bank' -d 'root@tcp(127.0.0.1:3306)/bank' --test-round=20000 -a 127.0.0.1:20080 go: downloading go1.23.4 (linux/amd64) table new_ci_collation_test.t2 exists table new_ci_collation_test.t3 not exists for 1-th check, retry later wait process cdc.test exit for 6-th time... table cdc_tiflash_test.multi_data_type exists run task successfully check_changefeed_state http://127.0.0.1:2379 0d49e698-cb6d-486a-90c9-5ab64d307a8d stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=0d49e698-cb6d-486a-90c9-5ab64d307a8d + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 0d49e698-cb6d-486a-90c9-5ab64d307a8d -s TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later + info='{ check diff failed 1-th time, retry later "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "stopped", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null }' + echo '{ "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "stopped", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null }' { "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "stopped", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null } ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"0d49e698-cb6d-486a-90c9-5ab64d307a8d",' '"state":' '"stopped",' '"checkpoint_tso":' 457114187507171332, '"checkpoint_time":' '"2025-04-04' '15:37:21.162",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"0d49e698-cb6d-486a-90c9-5ab64d307a8d",' '"state":' '"stopped",' '"checkpoint_tso":' 457114187507171332, '"checkpoint_time":' '"2025-04-04' '15:37:21.162",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489358772098331220 wait process cdc.test exit for 2-th time... wait process cdc.test exit for 7-th time... check diff successfully [Fri Apr 4 15:37:22 CST 2025] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.29302932.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:22 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458 {"id":"7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752240} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb6285e8 7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458 /tidb/cdc/default/default/upstream/7489358790029552117 {"id":7489358790029552117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458 {"id":"7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752240} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb6285e8 7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458 /tidb/cdc/default/default/upstream/7489358790029552117 {"id":7489358790029552117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458 {"id":"7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752240} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb6285e8 7bbb3ef0-eee0-4dd5-bad5-31ea52fd2458 /tidb/cdc/default/default/upstream/7489358790029552117 {"id":7489358790029552117,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2828.out cli changefeed create --start-ts=457114186762682369 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ table clustered_index_test.t2 exists check table exists success check diff failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:23 CST 2025] <<<<<< run test case changefeed_fast_fail success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Create changefeed successfully! ID: c26cb0dd-6bce-458c-ad28-e42c22e1cbb6 Info: {"upstream_id":7489358790029552117,"namespace":"default","id":"c26cb0dd-6bce-458c-ad28-e42c22e1cbb6","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:23.324886911+08:00","start_ts":457114186762682369,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114186762682369,"checkpoint_ts":457114186762682369,"checkpoint_time":"2025-04-04 15:37:18.322"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.11284.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 8-th time... owner pid: 2749 owner id 2e213e98-2124-4b4c-8d33-4c788b85d3d1 [Fri Apr 4 15:37:23 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.29072909.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 36-th check, retry later cdc.test: no process found wait process cdc.test exit for 9-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:37:23 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.59585960.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table new_ci_collation_test.t3 exists table new_ci_collation_test.t4 not exists for 1-th check, retry later check data successfully + set +x go: downloading go1.23.4 (linux/amd64) wait process cdc.test exit for 1-th time... + set +x + tso='457114188121112577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114188121112577 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114188121112577 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:37:24 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1132411326.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/pingcap/tidb/parser v0.0.0-20211122065349-4d16aabf900b go: downloading github.com/go-sql-driver/mysql v1.6.0 go: downloading github.com/pingcap/errors v0.11.5-0.20211009033009-93128226aaa3 go: downloading github.com/sirupsen/logrus v1.8.1 go: downloading go.uber.org/atomic v1.9.0 go: downloading golang.org/x/sys v0.0.0-20211117180635-dee7805ff2e1 go: downloading go.uber.org/zap v1.19.1 go: downloading github.com/pingcap/log v0.0.0-20210906054005-afc726e70354 go: downloading golang.org/x/text v0.3.7 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 go: downloading go.uber.org/multierr v1.7.0 check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/b07e96df-5865-4319-8f11-66d85e2da821 {UpstreamID:7489358773118019169 Namespace:default ID:b07e96df-5865-4319-8f11-66d85e2da821 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:16.415023756 +0800 CST StartTs:457114186234200066 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037e7440 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186260414469} {CheckpointTs:457114188357566468 MinTableBarrierTs:457114188619710468 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/45d6dac5-9eec-4cea-9b4d-fe4f392a46e6 {"id":"45d6dac5-9eec-4cea-9b4d-fe4f392a46e6","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752242} /tidb/cdc/default/__cdc_meta__/capture/dc11d138-27ca-4b40-a40a-2f803fd9a1fb {"id":"dc11d138-27ca-4b40-a40a-2f803fd9a1fb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501bd3 dc11d138-27ca-4b40-a40a-2f803fd9a1fb /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501c71 45d6dac5-9eec-4cea-9b4d-fe4f392a46e6 /tidb/cdc/default/default/changefeed/info/b07e96df-5865-4319-8f11-66d85e2da821 {"upstream-id":7489358773118019169,"namespace":"default","changefeed-id":"b07e96df-5865-4319-8f11-66d85e2da821","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:16.415023756+08:00","start-ts":457114186234200066,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186260414469} /tidb/cdc/default/default/changefeed/status/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":457114188357566468,"min-table-barrier-ts":457114188619710468,"admin-job-type":0} /tidb/cdc/default/default/task/position/45d6dac5-9eec-4cea-9b4d-fe4f392a46e6/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/dc11d138-27ca-4b40-a40a-2f803fd9a1fb/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358773118019169 {"id":7489358773118019169,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/b07e96df-5865-4319-8f11-66d85e2da821 {UpstreamID:7489358773118019169 Namespace:default ID:b07e96df-5865-4319-8f11-66d85e2da821 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:16.415023756 +0800 CST StartTs:457114186234200066 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037e7440 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186260414469} {CheckpointTs:457114188357566468 MinTableBarrierTs:457114188619710468 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/45d6dac5-9eec-4cea-9b4d-fe4f392a46e6 {"id":"45d6dac5-9eec-4cea-9b4d-fe4f392a46e6","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752242} /tidb/cdc/default/__cdc_meta__/capture/dc11d138-27ca-4b40-a40a-2f803fd9a1fb {"id":"dc11d138-27ca-4b40-a40a-2f803fd9a1fb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501bd3 dc11d138-27ca-4b40-a40a-2f803fd9a1fb /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501c71 45d6dac5-9eec-4cea-9b4d-fe4f392a46e6 /tidb/cdc/default/default/changefeed/info/b07e96df-5865-4319-8f11-66d85e2da821 {"upstream-id":7489358773118019169,"namespace":"default","changefeed-id":"b07e96df-5865-4319-8f11-66d85e2da821","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:16.415023756+08:00","start-ts":457114186234200066,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186260414469} /tidb/cdc/default/default/changefeed/status/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":457114188357566468,"min-table-barrier-ts":457114188619710468,"admin-job-type":0} /tidb/cdc/default/default/task/position/45d6dac5-9eec-4cea-9b4d-fe4f392a46e6/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/dc11d138-27ca-4b40-a40a-2f803fd9a1fb/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358773118019169 {"id":7489358773118019169,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/b07e96df-5865-4319-8f11-66d85e2da821 {UpstreamID:7489358773118019169 Namespace:default ID:b07e96df-5865-4319-8f11-66d85e2da821 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:16.415023756 +0800 CST StartTs:457114186234200066 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037e7440 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186260414469} {CheckpointTs:457114188357566468 MinTableBarrierTs:457114188619710468 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/45d6dac5-9eec-4cea-9b4d-fe4f392a46e6 {"id":"45d6dac5-9eec-4cea-9b4d-fe4f392a46e6","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752242} /tidb/cdc/default/__cdc_meta__/capture/dc11d138-27ca-4b40-a40a-2f803fd9a1fb {"id":"dc11d138-27ca-4b40-a40a-2f803fd9a1fb","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752233} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501bd3 dc11d138-27ca-4b40-a40a-2f803fd9a1fb /tidb/cdc/default/__cdc_meta__/owner/223195ffbb501c71 45d6dac5-9eec-4cea-9b4d-fe4f392a46e6 /tidb/cdc/default/default/changefeed/info/b07e96df-5865-4319-8f11-66d85e2da821 {"upstream-id":7489358773118019169,"namespace":"default","changefeed-id":"b07e96df-5865-4319-8f11-66d85e2da821","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:16.415023756+08:00","start-ts":457114186234200066,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186260414469} /tidb/cdc/default/default/changefeed/status/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":457114188357566468,"min-table-barrier-ts":457114188619710468,"admin-job-type":0} /tidb/cdc/default/default/task/position/45d6dac5-9eec-4cea-9b4d-fe4f392a46e6/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/dc11d138-27ca-4b40-a40a-2f803fd9a1fb/b07e96df-5865-4319-8f11-66d85e2da821 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358773118019169 {"id":7489358773118019169,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:25 CST 2025] <<<<<< run test case simple success! >>>>>> run task successfully table new_ci_collation_test.t4 exists table new_ci_collation_test.t5 not exists for 1-th check, retry later check_changefeed_state http://127.0.0.1:2379 0d49e698-cb6d-486a-90c9-5ab64d307a8d normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=0d49e698-cb6d-486a-90c9-5ab64d307a8d + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 0d49e698-cb6d-486a-90c9-5ab64d307a8d -s + info='{ "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "normal", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null }' + echo '{ "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "normal", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null }' { "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "normal", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null } ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"0d49e698-cb6d-486a-90c9-5ab64d307a8d",' '"state":' '"normal",' '"checkpoint_tso":' 457114187507171332, '"checkpoint_time":' '"2025-04-04' '15:37:21.162",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"0d49e698-cb6d-486a-90c9-5ab64d307a8d",' '"state":' '"normal",' '"checkpoint_tso":' 457114187507171332, '"checkpoint_time":' '"2025-04-04' '15:37:21.162",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7489358772098331220 457114187507171331 457114187507171332 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:26 CST 2025] <<<<<< run test case tiflash success! >>>>>> table ddl_manager.finish_mark not exists for 37-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading google.golang.org/grpc v1.64.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/google/btree v1.1.2 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading golang.org/x/net v0.33.0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:26 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0044181b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114185889218580 MinTableBarrierTs:457114188733480970 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114185889218580, checkpointTs: 457114185889218580, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e213e98-2124-4b4c-8d33-4c788b85d3d1 {"id":"2e213e98-2124-4b4c-8d33-4c788b85d3d1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/capture/4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 {"id":"4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752243} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb09 2e213e98-2124-4b4c-8d33-4c788b85d3d1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb70 4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114185889218580,"min-table-barrier-ts":457114188733480970,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e213e98-2124-4b4c-8d33-4c788b85d3d1/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0044181b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114185889218580 MinTableBarrierTs:457114188733480970 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114185889218580, checkpointTs: 457114185889218580, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e213e98-2124-4b4c-8d33-4c788b85d3d1 {"id":"2e213e98-2124-4b4c-8d33-4c788b85d3d1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/capture/4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 {"id":"4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752243} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb09 2e213e98-2124-4b4c-8d33-4c788b85d3d1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb70 4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114185889218580,"min-table-barri+ grep -q 'failed to get info:' er-ts":457114188733480970,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e213e98-2124-4b4c-8d33-4c788b85d3d1/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0044181b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114185889218580 MinTableBarrierTs:457114188733480970 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114185889218580, checkpointTs: 457114185889218580, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e213e98-2124-4b4c-8d33-4c788b85d3d1 {"id":"2e213e98-2124-4b4c-8d33-4c788b85d3d1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752235} /tidb/cdc/default/__cdc_meta__/capture/4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 {"id":"4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752243} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb09 2e213e98-2124-4b4c-8d33-4c788b85d3d1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdb70 4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114185889218580,"min-table-barrier-ts":457114188733480970,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e213e98-2124-4b4c-8d33-4c788b85d3d1/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "2e213e98-2124-4b4c-8d33-4c788b85d3d1" | grep -v cluster_id | grep id go: downloading golang.org/x/text v0.21.0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:27 GMT < Content-Length: 835 < Content-Type: text/plain; charset=utf-8 < { [835 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e15a11dd-9a71-481c-a0fd-f73091cb2078 {"id":"e15a11dd-9a71-481c-a0fd-f73091cb2078","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752244} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96f72 e15a11dd-9a71-481c-a0fd-f73091cb2078 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e15a11dd-9a71-481c-a0fd-f73091cb2078 {"id":"e15a11dd-9a71-481c-a0fd-f73091cb2078","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752244} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96f72 e15a11dd-9a71-481c-a0fd-f73091cb2078 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e15a11dd-9a71-481c-a0fd-f73091cb2078 {"id":"e15a11dd-9a71-481c-a0fd-f73091cb2078","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752244} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96f72 e15a11dd-9a71-481c-a0fd-f73091cb2078 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6011.out cli changefeed create --start-ts=457114178622586885 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-1 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading go.uber.org/zap v1.27.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.uber.org/multierr v1.11.0 go: downloading golang.org/x/sys v0.28.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading google.golang.org/protobuf v1.35.2 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda run task successfully check_changefeed_state http://127.0.0.1:2379 0d49e698-cb6d-486a-90c9-5ab64d307a8d stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=0d49e698-cb6d-486a-90c9-5ab64d307a8d + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 0d49e698-cb6d-486a-90c9-5ab64d307a8d -s check diff failed 3-th time, retry later Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7489358661568166943,"namespace":"default","id":"changefeed-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:37:27.699513214+08:00","start_ts":457114178622586885,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114178622586885,"checkpoint_ts":457114178622586885,"checkpoint_time":"2025-04-04 15:36:47.270"} PASS + info='{ "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "stopped", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null }' + echo '{ "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "stopped", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null }' { "upstream_id": 7489358772098331220, "namespace": "default", "id": "0d49e698-cb6d-486a-90c9-5ab64d307a8d", "state": "stopped", "checkpoint_tso": 457114187507171332, "checkpoint_time": "2025-04-04 15:37:21.162", "error": null } ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"0d49e698-cb6d-486a-90c9-5ab64d307a8d",' '"state":' '"stopped",' '"checkpoint_tso":' 457114187507171332, '"checkpoint_time":' '"2025-04-04' '15:37:21.162",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"0d49e698-cb6d-486a-90c9-5ab64d307a8d",' '"state":' '"stopped",' '"checkpoint_tso":' 457114187507171332, '"checkpoint_time":' '"2025-04-04' '15:37:21.162",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state http://127.0.0.1:2379 b5b388fa-5676-4b7c-843e-ed1e3642aa2d normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=b5b388fa-5676-4b7c-843e-ed1e3642aa2d + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c b5b388fa-5676-4b7c-843e-ed1e3642aa2d -s coverage: 2.4% of statements in github.com/pingcap/tiflow/... "cluster-id": "default" "id": "4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4", "cluster-id": "default" run task successfully table new_ci_collation_test.t5 exists check diff failed 1-th time, retry later + info='{ "upstream_id": 7489358772098331220, "namespace": "default", "id": "b5b388fa-5676-4b7c-843e-ed1e3642aa2d", "state": "normal", "checkpoint_tso": 457114189237321732, "checkpoint_time": "2025-04-04 15:37:27.762", "error": null }' + echo '{ "upstream_id": 7489358772098331220, "namespace": "default", "id": "b5b388fa-5676-4b7c-843e-ed1e3642aa2d", "state": "normal", "checkpoint_tso": 457114189237321732, "checkpoint_time": "2025-04-04 15:37:27.762", "error": null }' { "upstream_id": 7489358772098331220, "namespace": "default", "id": "b5b388fa-5676-4b7c-843e-ed1e3642aa2d", "state": "normal", "checkpoint_tso": 457114189237321732, "checkpoint_time": "2025-04-04 15:37:27.762", "error": null } ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"b5b388fa-5676-4b7c-843e-ed1e3642aa2d",' '"state":' '"normal",' '"checkpoint_tso":' 457114189237321732, '"checkpoint_time":' '"2025-04-04' '15:37:27.762",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358772098331220, '"namespace":' '"default",' '"id":' '"b5b388fa-5676-4b7c-843e-ed1e3642aa2d",' '"state":' '"normal",' '"checkpoint_tso":' 457114189237321732, '"checkpoint_time":' '"2025-04-04' '15:37:27.762",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7489358772098331220 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:28 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb {"id":"14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752245} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb7aade4 14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb /tidb/cdc/default/default/upstream/7489358816165049312 {"id":7489358816165049312,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb {"id":"14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752245} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb7aade4 14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb /tidb/cdc/default/default/upstream/7489358816165049312 {"id":7489358816165049312,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb {"id":"14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752245} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb7aade4 14d6e378-2a5c-4f74-8a8f-f2f0fdff8afb /tidb/cdc/default/default/upstream/7489358816165049312 {"id":7489358816165049312,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114188121112577 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.11382.out cli changefeed create --start-ts=457114188121112577 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml table ddl_manager.finish_mark not exists for 38-th check, retry later Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489358816165049312,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:37:28.503375927+08:00","start_ts":457114188121112577,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114188121112577,"checkpoint_ts":457114188121112577,"checkpoint_time":"2025-04-04 15:37:23.504"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/stretchr/testify v1.10.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/golang/snappy v0.0.4 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 check diff failed 1-th time, retry later go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/spf13/cobra v1.8.1 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading k8s.io/api v0.29.11 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading cloud.google.com/go/iam v1.1.7 capture_id: 4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 check diff failed 4-th time, retry later go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading cloud.google.com/go/kms v1.15.8 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 wait process 2749 exit for 1-th time... go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 6314 0 --:--:-- --:--:-- --:--:-- 6314 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:37:23.504","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:37:29.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:37:23.504","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:37:29.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:37:23.504","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:37:29.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:37:23.504' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:37:23.504","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:37:29.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:37:23.504","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:37:29.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:37:30' + echo 'sink_checkpoint_ts is 2025-04-04' 15:37:23.504 sink_checkpoint_ts is 2025-04-04 15:37:23.504 ++ date -d '2025-04-04 15:37:23.504' +%s + checkpoint_timestamp=1743752243 ++ date -d '2025-04-04 15:37:30' +%s + current_timestamp=1743752250 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 2811 0 --:--:-- --:--:-- --:--:-- 2780 100 253 100 253 0 0 2811 0 --:--:-- --:--:-- --:--:-- 2780 + info='{"state":"warning","resolved_ts":457114179015802883,"checkpoint_ts":457114179015802883,"last_warning":{"time":"2025-04-04T15:37:29.877253353+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":457114179015802883,"checkpoint_ts":457114179015802883,"last_warning":{"time":"2025-04-04T15:37:29.877253353+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":457114179015802883,"checkpoint_ts":457114179015802883,"last_warning":{"time":"2025-04-04T15:37:29.877253353+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":457114179015802883,"checkpoint_ts":457114179015802883,"last_warning":{"time":"2025-04-04T15:37:29.877253353+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":457114179015802883,"checkpoint_ts":457114179015802883,"last_warning":{"time":"2025-04-04T15:37:29.877253353+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6055.out cli changefeed remove -c changefeed-error-1 wait process 2749 exit for 2-th time... check diff failed 2-th time, retry later go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading k8s.io/apimachinery v0.29.11 + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 table ddl_manager.finish_mark not exists for 39-th check, retry later Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 457114179015802883 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2749) - No such process wait process 2749 exit for 3-th time... process 2749 already exit cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 |grep 4ed23a7a-5df7-4f9c-a32e-bfc4863a44c4 -A1 | grep '"is-owner": true' go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b coverage: 2.0% of statements in github.com/pingcap/tiflow/... run task successfully check diff failed 5-th time, retry later check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Changefeed remove successfully. ID: 0d49e698-cb6d-486a-90c9-5ab64d307a8d CheckpointTs: 457114187507171332 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_forward http://127.0.0.1:2379 7489358772098331220 457114190128611334 457114187507171332 457114189237321732 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible "is-owner": true, run task successfully test_kill_owner: pass go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/google/btree v1.1.2 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading github.com/gorilla/mux v1.8.1 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading golang.org/x/sync v0.10.0 go: downloading golang.org/x/net v0.33.0 go: downloading golang.org/x/sys v0.28.0 go: downloading google.golang.org/grpc v1.64.0 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading golang.org/x/text v0.21.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading cloud.google.com/go v0.112.2 go: downloading golang.org/x/oauth2 v0.24.0 check diff failed 3-th time, retry later table test.t1 exists + sleep 5 + set +x go: downloading google.golang.org/api v0.170.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading golang.org/x/time v0.7.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading google.golang.org/protobuf v1.35.2 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/dgraph-io/ristretto v0.1.1 table ddl_manager.finish_mark not exists for 40-th check, retry later run task successfully go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 wait process cdc.test exit for 1-th time... go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/batch_update_to_no_batch/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... Changefeed remove successfully. ID: b5b388fa-5676-4b7c-843e-ed1e3642aa2d CheckpointTs: 457114190390755336 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_cleared http://127.0.0.1:2379 7489358772098331220 run task successfully go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/coreos/go-systemd/v22 v22.5.0 check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 wait process cdc.test exit for 2-th time... go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading golang.org/x/term v0.27.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible wait process cdc.test exit for 2-th time... go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 wait process cdc.test exit for 1-th time... go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da check diff failed 6-th time, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_hang_up_owner [Fri Apr 4 15:37:33 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.30863088.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/kr/text v0.2.0 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 4 15:37:34 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.61186120.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 41-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:34 CST 2025] <<<<<< run test case gc_safepoint success! >>>>>> go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff failed 7-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358880650040547 TiDB Cluster ID. tikv_gc_leader_uuid 657feef464c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:9312, start at 2025-04-04 15:37:33.847282805 +0800 CST m=+2.224774287 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:33.854 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:33.843 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:33.843 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358880650040547 TiDB Cluster ID. tikv_gc_leader_uuid 657feef464c0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:9312, start at 2025-04-04 15:37:33.847282805 +0800 CST m=+2.224774287 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:33.854 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:33.843 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:33.843 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358873449457171 TiDB Cluster ID. tikv_gc_leader_uuid 657feef4640000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-ftv0p-w18v8-qqq9b, pid:9387, start at 2025-04-04 15:37:33.88499171 +0800 CST m=+2.194411845 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:33.893 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:33.890 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:33.890 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading github.com/pierrec/lz4 v2.6.1+incompatible table ddl_manager.finish_mark not exists for 42-th check, retry later check diff failed 5-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:37 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004392e10 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114191354920967 MinTableBarrierTs:457114191617327112 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114191617327110, checkpointTs: 457114191617327110, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114191617327112, checkpointTs: 457114191617327112, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114191617327112, checkpointTs: 457114191617327112, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f67b3871-8c85-4d8e-ba11-b1d0cbc55156 {"id":"f67b3871-8c85-4d8e-ba11-b1d0cbc55156","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752254} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdbf0 f67b3871-8c85-4d8e-ba11-b1d0cbc55156 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114191354920967,"min-table-barrier-ts":457114191617327112,"admin-job-type":0} /tidb/cdc/default/default/task/position/f67b3871-8c85-4d8e-ba11-b1d0cbc55156/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004392e10 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114191354920967 MinTableBarrierTs:457114191617327112 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114191617327110, checkpointTs: 457114191617327110, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114191617327112, checkpointTs: 457114191617327112, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114191617327112, checkpointTs: 457114191617327112, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f67b3871-8c85-4d8e-ba11-b1d0cbc55156 {"id":"f67b3871-8c85-4d8e-ba11-b1d0cbc55156","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752254} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdbf0 f67b3871-8c85-4d8e-ba11-b1d0cbc55156 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 start tidb cluster in /tmp/tidb_cdc_test/batch_update_to_no_batch Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114191354920967,"min-table-barrier-ts":457114191617327112,"admin-job-type":0} /tidb/cdc/default/default/task/position/f67b3871-8c85-4d8e-ba11-b1d0cbc55156/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004392e10 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114191354920967 MinTableBarrierTs:457114191617327112 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114191617327110, checkpointTs: 457114191617327110, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114191617327112, checkpointTs: 457114191617327112, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114191617327112, checkpointTs: 457114191617327112, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f67b3871-8c85-4d8e-ba11-b1d0cbc55156 {"id":"f67b3871-8c85-4d8e-ba11-b1d0cbc55156","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752254} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdbf0 f67b3871-8c85-4d8e-ba11-b1d0cbc55156 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114191354920967,"min-table-barrier-ts":457114191617327112,"admin-job-type":0} /tidb/cdc/default/default/task/position/f67b3871-8c85-4d8e-ba11-b1d0cbc55156/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' check diff failed 8-th time, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 24300 0 --:--:-- --:--:-- --:--:-- 24300 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:37:35.754","puller_resolved_ts":"2025-04-04 15:37:30.003","last_synced_ts":"2025-04-04 15:37:30.504","now_ts":"2025-04-04 15:37:37.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:37:35.754","puller_resolved_ts":"2025-04-04' '15:37:30.003","last_synced_ts":"2025-04-04' '15:37:30.504","now_ts":"2025-04-04' '15:37:37.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:37:35.754","puller_resolved_ts":"2025-04-04' '15:37:30.003","last_synced_ts":"2025-04-04' '15:37:30.504","now_ts":"2025-04-04' '15:37:37.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:37 GMT < Content-Length: 835 < Content-Type: text/plain; charset=utf-8 < { [835 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a219ae14-b9be-426c-8c58-1fa85a754841 {"id":"a219ae14-b9be-426c-8c58-1fa85a754841","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752255} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96fee a219ae14-b9be-426c-8c58-1fa85a754841 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a219ae14-b9be-426c-8c58-1fa85a754841 {"id":"a219ae14-b9be-426c-8c58-1fa85a754841","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752255} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96fee a219ae14-b9be-426c-8c58-1fa85a754841 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a219ae14-b9be-426c-8c58-1fa85a754841 {"id":"a219ae14-b9be-426c-8c58-1fa85a754841","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752255} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae96fee a219ae14-b9be-426c-8c58-1fa85a754841 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6169.out cli changefeed create --start-ts=457114178622586885 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-2 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10620.out cli tso query --pd=http://127.0.0.1:2379 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/felixge/httpsnoop v1.0.4 Verifying downstream PD is started... Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7489358661568166943,"namespace":"default","id":"changefeed-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:37:38.282615032+08:00","start_ts":457114178622586885,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114178622586885,"checkpoint_ts":457114178622586885,"checkpoint_time":"2025-04-04 15:36:47.270"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 43-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading github.com/jmespath/go-jmespath v0.4.0 check diff successfully "is-owner": true, run task successfully start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:37:38.789 +08:00] [INFO] [case.go:450] ["open db success"] [dsn=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:37:38.790 +08:00] [INFO] [case.go:450] ["open db success"] [dsn=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:37:38.791 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq0] [2025/04/04 15:37:38.830 +08:00] [INFO] [case.go:370] ["insert 100 takes 3.538788ms"] [query="INSERT IGNORE INTO accounts_seq0 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:37:38.831 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq1] [2025/04/04 15:37:38.855 +08:00] [INFO] [case.go:370] ["insert 100 takes 3.692758ms"] [query="INSERT IGNORE INTO accounts_seq1 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:37:38.855 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq2] [2025/04/04 15:37:38.879 +08:00] [INFO] [case.go:370] ["insert 100 takes 3.423099ms"] [query="INSERT IGNORE INTO accounts_seq2 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:37:38.879 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq3] [2025/04/04 15:37:38.903 +08:00] [INFO] [case.go:370] ["insert 100 takes 5.234925ms"] [query="INSERT IGNORE INTO accounts_seq3 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:37:38.904 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts_seq4] [2025/04/04 15:37:38.927 +08:00] [INFO] [case.go:370] ["insert 100 takes 3.914032ms"] [query="INSERT IGNORE INTO accounts_seq4 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2025/04/04 15:37:38.927 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts0] [2025/04/04 15:37:38.953 +08:00] [INFO] [case.go:370] ["insert 100 takes 3.775511ms"] [query="INSERT IGNORE INTO accounts0 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:37:38.954 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts1] [2025/04/04 15:37:38.977 +08:00] [INFO] [case.go:370] ["insert 100 takes 3.913786ms"] [query="INSERT IGNORE INTO accounts1 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:37:38.977 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts2] [2025/04/04 15:37:39.001 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.068412ms"] [query="INSERT IGNORE INTO accounts2 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:37:39.001 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts3] [2025/04/04 15:37:39.025 +08:00] [INFO] [case.go:370] ["insert 100 takes 3.775737ms"] [query="INSERT IGNORE INTO accounts3 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2025/04/04 15:37:39.025 +08:00] [INFO] [case.go:388] ["drop tables"] [table=accounts4] [2025/04/04 15:37:39.050 +08:00] [INFO] [case.go:370] ["insert 100 takes 4.15046ms"] [query="INSERT IGNORE INTO accounts4 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] ***************** properties ***************** "workload"="core" "scanproportion"="0" "dotransactions"="false" "operationcount"="0" "readallfields"="true" "requestdistribution"="uniform" "mysql.port"="4000" "mysql.user"="root" "updateproportion"="0" "mysql.db"="testSync" "threadcount"="2" "mysql.host"="127.0.0.1" "readproportion"="0" "insertproportion"="0" "recordcount"="10" ********************************************** Run finished, takes 6.740792ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2257.7, Avg(us): 1073, Min(us): 451, Max(us): 3584, 95th(us): 4000, 99th(us): 4000 + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s + set +x + tso='457114192024174593 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114192024174593 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:37:39 CST 2025] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1066410666.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + info='{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114178622586885, "checkpoint_time": "2025-04-04 15:36:47.270", "error": { "time": "2025-04-04T15:37:38.369683754+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114178622586885 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114178622586885, "checkpoint_time": "2025-04-04 15:36:47.270", "error": { "time": "2025-04-04T15:37:38.369683754+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114178622586885 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 457114178622586885, "checkpoint_time": "2025-04-04 15:36:47.270", "error": { "time": "2025-04-04T15:37:38.369683754+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114178622586885 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114178622586885, '"checkpoint_time":' '"2025-04-04' '15:36:47.270",' '"error":' '{' '"time":' '"2025-04-04T15:37:38.369683754+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114178622586885 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114178622586885, '"checkpoint_time":' '"2025-04-04' '15:36:47.270",' '"error":' '{' '"time":' '"2025-04-04T15:37:38.369683754+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114178622586885 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114178622586885 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 457114178622586885 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6229.out cli changefeed remove -c changefeed-error-2 owner pid: 3091 owner id f67b3871-8c85-4d8e-ba11-b1d0cbc55156 [Fri Apr 4 15:37:39 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.31913193.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 9-th time, retry later table ddl_manager.finish_mark not exists for 44-th check, retry later Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 457114178622586885 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:37:41.073 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 10-th time, retry later + set +x table ddl_manager.finish_mark not exists for 45-th check, retry later wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:42 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78bf3632-fbd3-491e-98de-c523327195ad {"id":"78bf3632-fbd3-491e-98de-c523327195ad","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752260} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbb4d5e7 78bf3632-fbd3-491e-98de-c523327195ad /tidb/cdc/default/default/upstream/7489358880650040547 {"id":7489358880650040547,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78bf3632-fbd3-491e-98de-c523327195ad {"id":"78bf3632-fbd3-491e-98de-c523327195ad","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752260} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbb4d5e7 78bf3632-fbd3-491e-98de-c523327195ad /tidb/cdc/default/default/upstream/7489358880650040547 {"id":7489358880650040547,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/78bf3632-fbd3-491e-98de-c523327195ad {"id":"78bf3632-fbd3-491e-98de-c523327195ad","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752260} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbb4d5e7 78bf3632-fbd3-491e-98de-c523327195ad /tidb/cdc/default/default/upstream/7489358880650040547 {"id":7489358880650040547,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10719.out cli changefeed create --start-ts=457114192024174593 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:43 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00393e900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114192928047115 MinTableBarrierTs:457114193189928974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6c57b7d-5a1f-4757-868a-a5c7f2e2562f {"id":"b6c57b7d-5a1f-4757-868a-a5c7f2e2562f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752260} /tidb/cdc/default/__cdc_meta__/capture/f67b3871-8c85-4d8e-ba11-b1d0cbc55156 {"id":"f67b3871-8c85-4d8e-ba11-b1d0cbc55156","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752254} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdbf0 f67b3871-8c85-4d8e-ba11-b1d0cbc55156 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdc59 b6c57b7d-5a1f-4757-868a-a5c7f2e2562f /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114192928047115,"min-table-barrier-ts":457114193189928974,"admin-job-type":0} /tidb/cdc/default/default/task/position/b6c57b7d-5a1f-4757-868a-a5c7f2e2562f/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f67b3871-8c85-4d8e-ba11-b1d0cbc55156/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00393e900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114192928047115 MinTableBarrierTs:457114193189928974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6c57b7d-5a1f-4757-868a-a5c7f2e2562f {"id":"b6c57b7d-5a1f-4757-868a-a5c7f2e2562f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752260} /tidb/cdc/default/__cdc_meta__/capture/f67b3871-8c85-4d8e-ba11-b1d0cbc55156 {"id":"f67b3871-8c85-4d8e-ba11-b1d0cbc55156","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752254} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdbf0 f67b3871-8c85-4d8e-ba11-b1d0cbc55156 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdc59 b6c57b7d-5a1f-4757-868a-a5c7f2e2562f /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114192928047115,"min-table-barrier-ts":457114193189928974,"admin-job-type":0} /tidb/cdc/default/default/task/position/b6c57b7d-5a1f-4757-868a-a5c7f2e2562f/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f67b3871-8c85-4d8e-ba11-b1d0cbc55156/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00393e900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114192928047115 MinTableBarrierTs:457114193189928974 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6c57b7d-5a1f-4757-868a-a5c7f2e2562f {"id":"b6c57b7d-5a1f-4757-868a-a5c7f2e2562f","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752260} /tidb/cdc/default/__cdc_meta__/capture/f67b3871-8c85-4d8e-ba11-b1d0cbc55156 {"id":"f67b3871-8c85-4d8e-ba11-b1d0cbc55156","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752254} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdbf0 f67b3871-8c85-4d8e-ba11-b1d0cbc55156 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdc59 b6c57b7d-5a1f-4757-868a-a5c7f2e2562f /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114192928047115,"min-table-barrier-ts":457114193189928974,"admin-job-type":0} /tidb/cdc/default/default/task/position/b6c57b7d-5a1f-4757-868a-a5c7f2e2562f/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f67b3871-8c85-4d8e-ba11-b1d0cbc55156/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "f67b3871-8c85-4d8e-ba11-b1d0cbc55156" | grep id check diff successfully wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:37:43.078 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] Create changefeed successfully! ID: 57516d1d-5ee3-4302-a639-8434032f6893 Info: {"upstream_id":7489358880650040547,"namespace":"default","id":"57516d1d-5ee3-4302-a639-8434032f6893","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:43.423123147+08:00","start_ts":457114192024174593,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114192024174593,"checkpoint_ts":457114192024174593,"checkpoint_time":"2025-04-04 15:37:38.393"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:43 CST 2025] <<<<<< START cdc server in changefeed_error case >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.62846286.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 11-th time, retry later check diff successfully wait process cdc.test exit for 2-th time... "cluster-id": "default" "id": "b6c57b7d-5a1f-4757-868a-a5c7f2e2562f", "cluster-id": "default" run task successfully wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 46-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:44 CST 2025] <<<<<< run test case owner_resign success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x wait process cdc.test exit for 2-th time... [2025/04/04 15:37:45.083 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] wait process cdc.test exit for 3-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 12-th time, retry later capture_id: b6c57b7d-5a1f-4757-868a-a5c7f2e2562f ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep 'b6c57b7d-5a1f-4757-868a-a5c7f2e2562f' b6c57b7d-5a1f-4757-868a-a5c7f2e2562f run task successfully test_hang_up_owner: pass ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 4-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358918194230688 TiDB Cluster ID. tikv_gc_leader_uuid 657feefe4840002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:3942, start at 2025-04-04 15:37:43.970999015 +0800 CST m=+2.120260357 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:43.978 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:43.969 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:43.969 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358918194230688 TiDB Cluster ID. tikv_gc_leader_uuid 657feefe4840002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:3942, start at 2025-04-04 15:37:43.970999015 +0800 CST m=+2.120260357 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:43.978 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:43.969 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:43.969 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358920933745867 TiDB Cluster ID. tikv_gc_leader_uuid 657feefe608000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:4014, start at 2025-04-04 15:37:44.08399736 +0800 CST m=+2.136002496 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:44.092 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:44.066 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:44.066 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... wait process cdc.test exit for 5-th time... table ddl_manager.finish_mark not exists for 47-th check, retry later build mysql_test successfully run all mysql test cases time="2025-04-04T15:37:43+08:00" level=info msg="running tests: [alter_table bigint composite_index concurrent_ddl date_formats drop mysql_replace]" time="2025-04-04T15:37:43+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000134000 {{} {} 0} {0 0} [0xc000146080] {[]} 1 0xc000138000 false map[0xc000146080:map[0xc000146080:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 6-th time... process cdc.test already exit [Fri Apr 4 15:37:47 CST 2025] <<<<<< run test case new_ci_collation success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:46 GMT < Content-Length: 835 < Content-Type: text/plain; charset=utf-8 < { [835 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f4ad721b-985f-4402-a4b7-cee5c8ade3d3 {"id":"f4ad721b-985f-4402-a4b7-cee5c8ade3d3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752264} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae9706e f4ad721b-985f-4402-a4b7-cee5c8ade3d3 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f4ad721b-985f-4402-a4b7-cee5c8ade3d3 {"id":"f4ad721b-985f-4402-a4b7-cee5c8ade3d3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752264} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae9706e f4ad721b-985f-4402-a4b7-cee5c8ade3d3 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f4ad721b-985f-4402-a4b7-cee5c8ade3d3 {"id":"f4ad721b-985f-4402-a4b7-cee5c8ade3d3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752264} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 7 /tidb/cdc/default/__cdc_meta__/owner/223195ffbae9706e f4ad721b-985f-4402-a4b7-cee5c8ade3d3 /tidb/cdc/default/default/upstream/7489358661568166943 {"id":7489358661568166943,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6336.out cli changefeed create --start-ts=0 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-initialize-error [2025/04/04 15:37:47.089 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7489358661568166943,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:37:47.474194685+08:00","start_ts":457114194351226881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114194351226881,"checkpoint_ts":457114194351226881,"checkpoint_time":"2025-04-04 15:37:47.270"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_expire_owner [Fri Apr 4 15:37:47 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.33423344.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_owner.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358931240242907 TiDB Cluster ID. tikv_gc_leader_uuid 657fef0007c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:3843, start at 2025-04-04 15:37:45.779034012 +0800 CST m=+2.290343305 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:45.820 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:45.817 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:45.817 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358931240242907 TiDB Cluster ID. tikv_gc_leader_uuid 657fef0007c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:3843, start at 2025-04-04 15:37:45.779034012 +0800 CST m=+2.290343305 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:45.820 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:45.817 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:45.817 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358931300299997 TiDB Cluster ID. tikv_gc_leader_uuid 657fef001fc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:3910, start at 2025-04-04 15:37:45.885230323 +0800 CST m=+2.313262427 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:45.927 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:45.905 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:45.905 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 13-th time, retry later table ddl_manager.finish_mark not exists for 48-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114194351226881, "checkpoint_time": "2025-04-04 15:37:47.270", "error": { "time": "2025-04-04T15:37:47.651737915+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114194351226881, "checkpoint_time": "2025-04-04 15:37:47.270", "error": { "time": "2025-04-04T15:37:47.651737915+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 457114194351226881, "checkpoint_time": "2025-04-04 15:37:47.270", "error": { "time": "2025-04-04T15:37:47.651737915+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 457114194351226881, '"checkpoint_time":' '"2025-04-04' '15:37:47.270",' '"error":' '{' '"time":' '"2025-04-04T15:37:47.651737915+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later table partition_table.t exists table partition_table.t1 not exists for 1-th check, retry later [2025/04/04 15:37:49.095 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.5184.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 14-th time, retry later table ddl_manager.finish_mark not exists for 49-th check, retry later [Fri Apr 4 15:37:50 CST 2025] <<<<<< START cdc server in batch_update_to_no_batch case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.52905292.out server --log-file /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:50 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d6e240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114195038306306 MinTableBarrierTs:457114195038306308 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114195038306308, checkpointTs: 457114195038306308, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114195038306308, checkpointTs: 457114195038306308, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114195038306306, checkpointTs: 457114195038306306, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8a7e5b1-3584-48b2-bca0-0185607527d9 {"id":"f8a7e5b1-3584-48b2-bca0-0185607527d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752267} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdcc0 f8a7e5b1-3584-48b2-bca0-0185607527d9 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114195038306306,"min-table-barrier-ts":457114195038306308,"admin-job-type":0} /tidb/cdc/default/default/task/position/f8a7e5b1-3584-48b2-bca0-0185607527d9/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d6e240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114195038306306 MinTableBarrierTs:457114195038306308 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114195038306308, checkpointTs: 457114195038306308, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114195038306308, checkpointTs: 457114195038306308, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114195038306306, checkpointTs: 457114195038306306, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8a7e5b1-3584-48b2-bca0-0185607527d9 {"id":"f8a7e5b1-3584-48b2-bca0-0185607527d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752267} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdcc0 f8a7e5b1-3584-48b2-bca0-0185607527d9 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114195038306306,"min-table-barrier-ts":457114195038306308,"admin-job-type":0} /tidb/cdc/default/default/task/position/f8a7e5b1-3584-48b2-bca0-0185607527d9/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d6e240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114195038306306 MinTableBarrierTs:457114195038306308 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114195038306308, checkpointTs: 457114195038306308, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114195038306308, checkpointTs: 457114195038306308, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114195038306306, checkpointTs: 457114195038306306, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f8a7e5b1-3584-48b2-bca0-0185607527d9 {"id":"f8a7e5b1-3584-48b2-bca0-0185607527d9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752267} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdcc0 f8a7e5b1-3584-48b2-bca0-0185607527d9 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114195038306306,"min-table-barrier-ts":457114195038306308,"admin-job-type":0} /tidb/cdc/default/default/task/position/f8a7e5b1-3584-48b2-bca0-0185607527d9/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2025/04/04 15:37:51.101 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null }' + echo '{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null }' { "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null } ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114195242254337, '"checkpoint_time":' '"2025-04-04' '15:37:50.669",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114195242254337, '"checkpoint_time":' '"2025-04-04' '15:37:50.669",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6434.out cli changefeed pause -c changefeed-initialize-error + set +x + tso='457114195121668097 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114195121668097 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358947017772680 TiDB Cluster ID. tikv_gc_leader_uuid 657fef045980002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:3902, start at 2025-04-04 15:37:50.183507953 +0800 CST m=+2.939463573 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:50.191 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:50.182 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:50.182 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358947017772680 TiDB Cluster ID. tikv_gc_leader_uuid 657fef045980002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:3902, start at 2025-04-04 15:37:50.183507953 +0800 CST m=+2.939463573 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:50.191 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:50.182 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:50.182 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358938709542203 TiDB Cluster ID. tikv_gc_leader_uuid 657fef04024000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:3979, start at 2025-04-04 15:37:49.882468853 +0800 CST m=+2.542832818 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:49.931 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:49.932 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:49.932 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 15-th time, retry later try a VALID cdc server command [Fri Apr 4 15:37:52 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.52295231.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 "is-owner": true, run task successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 50-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_ddl/sync_diff/output" source-instances = ["tidb"] target-instance = "mysql" target-check-tables = ["consistent_replicate_ddl.usertable*","consistent_replicate_ddl.check*"] [data-sources] [data-sources.tidb] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "457114160427171846" [data-sources.mysql] host = "127.0.0.1" port = 3306 user = "root" password = "" [2025/04/04 15:37:40.613 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:37:40.614 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:37:40.614 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:37:40.614 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:37:40.614 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114158723235867] [resolvedTs=457114160427171846] [2025/04/04 15:37:40.615 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:37:40.615 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:37:40.615 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:37:40.616 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=2.041762ms] [2025/04/04 15:37:40.618 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=3.517944ms] [2025/04/04 15:37:40.620 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:37:40.623 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:37:40.624 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:37:40.624 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:37:40.625 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:37:40.625 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:37:40.629 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:37:40.631 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:37:40.631 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:37:40.631 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114158723235863,\"CommitTs\":457114158723235867,\"Query\":\"DROP TABLE `consistent_replicate_ddl`.`usertable1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable1\",\"TableID\":116,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":4,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:40.632 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:40.632 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114158723235867] [DDL="DROP TABLE `consistent_replicate_ddl`.`usertable1`"] [2025/04/04 15:37:40.704 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=71.350618ms] [sql="DROP TABLE `consistent_replicate_ddl`.`usertable1`"] [2025/04/04 15:37:40.704 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114158854570015,\"CommitTs\":457114158854570019,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable1\",\"TableID\":122,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:40.704 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:40.704 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114158854570019] [DDL="RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1"] [2025/04/04 15:37:40.742 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=37.940127ms] [sql="RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1"] [2025/04/04 15:37:40.742 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159011594252,\"CommitTs\":457114159011594255,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable1_1\",\"TableID\":122,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:40.944 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:40.944 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159011594255] [DDL="RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1"] [2025/04/04 15:37:40.984 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=40.187743ms] [sql="RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1"] [2025/04/04 15:37:40.984 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159155773451,\"CommitTs\":457114159155773454,\"Query\":\"CREATE DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl1\",\"Table\":\"\",\"TableID\":0,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":1,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.185 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159155773454] [DDL="CREATE DATABASE consistent_replicate_ddl1"] [2025/04/04 15:37:41.212 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=26.441352ms] [sql="CREATE DATABASE consistent_replicate_ddl1"] [2025/04/04 15:37:41.212 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159286845461,\"CommitTs\":457114159299952642,\"Query\":\"ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"\",\"TableID\":0,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":26,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.212 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:41.212 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159299952642] [DDL="ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:37:41.239 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=27.126787ms] [sql="ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:37:41.240 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159431024658,\"CommitTs\":457114159431024663,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.240 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:41.240 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159431024663] [DDL="ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:37:41.265 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=24.502125ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [2025/04/04 15:37:41.265 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159588311068,\"CommitTs\":457114159588311072,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":6,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.466 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159588311072] [DDL="ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0"] [2025/04/04 15:37:41.522 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=56.463102ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0"] [2025/04/04 15:37:41.522 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159732490248,\"CommitTs\":457114159732490251,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":5,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.522 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159732490251] [DDL="ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)"] [2025/04/04 15:37:41.587 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=64.467199ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)"] [2025/04/04 15:37:41.587 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159863562264,\"CommitTs\":457114159863562268,\"Query\":\"DROP DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl1\",\"Table\":\"\",\"TableID\":0,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":2,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.587 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159863562268] [DDL="DROP DATABASE consistent_replicate_ddl1"] [2025/04/04 15:37:41.633 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=45.920337ms] [sql="DROP DATABASE consistent_replicate_ddl1"] [2025/04/04 15:37:41.633 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114159994634258,\"CommitTs\":457114159994634262,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable3\",\"TableID\":120,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.634 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:41.634 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114159994634262] [DDL="ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"] [2025/04/04 15:37:41.656 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=22.737763ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"] [2025/04/04 15:37:41.657 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114160138813460,\"CommitTs\":457114160138813464,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable3_1\",\"TableID\":120,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.657 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:41.657 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114160138813464] [DDL="RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`"] [2025/04/04 15:37:41.694 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=36.641025ms] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`"] [2025/04/04 15:37:41.694 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114160138813460,\"CommitTs\":457114160138813464,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable2_1\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.695 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:37:41.695 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114160138813464] [DDL="RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`"] [2025/04/04 15:37:41.730 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=34.96879ms] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`"] [2025/04/04 15:37:41.730 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114160270147605,\"CommitTs\":457114160282992642,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"usertable3_1\",\"TableID\":120,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":12,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.730 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114160282992642] [DDL="ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)"] [2025/04/04 15:37:41.755 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=24.720076ms] [sql="ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)"] [2025/04/04 15:37:41.755 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114160427171843,\"CommitTs\":457114160427171846,\"Query\":\"CREATE table consistent_replicate_ddl.check1(id int primary key)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_ddl\",\"Table\":\"check1\",\"TableID\":137,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:37:41.955 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114160427171846] [DDL="CREATE table consistent_replicate_ddl.check1(id int primary key)"] [2025/04/04 15:37:41.982 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=26.513415ms] [sql="CREATE table consistent_replicate_ddl.check1(id int primary key)"] [2025/04/04 15:37:41.982 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114160427171846] [2025/04/04 15:37:41.982 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114160427171846] [2025/04/04 15:37:42.082 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114160427171846] [2025/04/04 15:37:42.082 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114160427171846] [2025/04/04 15:37:42.182 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}"] [checkpointTs=457114160427171846] [2025/04/04 15:37:42.182 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}"] [checkpointTs=457114160427171846] [2025/04/04 15:37:42.282 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=101] [appliedDDLCount=14] [currentCheckpoint=457114160427171846] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:37:42.283 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] Apply redo log successfully table consistent_replicate_ddl.check1 exists check diff successfully [Fri Apr 4 15:37:42 CST 2025] <<<<<< run test case consistent_replicate_ddl success! >>>>>> count(*) 50 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: mysql... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide table partition_table.t2 not exists for 2-th check, retry later [2025/04/04 15:37:53.105 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] + info='{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null }' + echo '{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null }' { "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null } ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114195242254337, '"checkpoint_time":' '"2025-04-04' '15:37:50.669",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 457114195242254337, '"checkpoint_time":' '"2025-04-04' '15:37:50.669",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6505.out cli changefeed resume -c changefeed-initialize-error owner pid: 3347 owner id f8a7e5b1-3584-48b2-bca0-0185607527d9 process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v 'f8a7e5b1-3584-48b2-bca0-0185607527d9' /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdcc0 run task successfully process status: S cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Fri Apr 4 15:37:53 CST 2025] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.52785280.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:53 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ba586e8-4e64-4720-915c-37cfce221e0f {"id":"0ba586e8-4e64-4720-915c-37cfce221e0f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752270} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbdc96ed 0ba586e8-4e64-4720-915c-37cfce221e0f /tidb/cdc/default/default/upstream/7489358918194230688 {"id":7489358918194230688,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ba586e8-4e64-4720-915c-37cfce221e0f {"id":"0ba586e8-4e64-4720-915c-37cfce221e0f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752270} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbdc96ed 0ba586e8-4e64-4720-915c-37cfce221e0f /tidb/cdc/default/default/upstream/7489358918194230688 {"id":7489358918194230688,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0ba586e8-4e64-4720-915c-37cfce221e0f {"id":"0ba586e8-4e64-4720-915c-37cfce221e0f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752270} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbdc96ed 0ba586e8-4e64-4720-915c-37cfce221e0f /tidb/cdc/default/default/upstream/7489358918194230688 {"id":7489358918194230688,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5342.out cli changefeed create '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' -c test Create changefeed successfully! ID: test Info: {"upstream_id":7489358918194230688,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2025-04-04T15:37:53.789880437+08:00","start_ts":457114196041793539,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114196041793539,"checkpoint_ts":457114196041793539,"checkpoint_time":"2025-04-04 15:37:53.719"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330D7468B7E2FB < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:37:54 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... check diff failed 16-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 51-th check, retry later "is-owner": true, run task successfully test_expire_owner pass + set +x [2025/04/04 15:37:55.111 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] table partition_table.t2 not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:55 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03adf792-3dd5-4b22-b4de-67c99245b415 {"id":"03adf792-3dd5-4b22-b4de-67c99245b415","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752272} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbe524f2 03adf792-3dd5-4b22-b4de-67c99245b415 /tidb/cdc/default/default/upstream/7489358931240242907 {"id":7489358931240242907,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03adf792-3dd5-4b22-b4de-67c99245b415 {"id":"03adf792-3dd5-4b22-b4de-67c99245b415","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752272} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbe524f2 03adf792-3dd5-4b22-b4de-67c99245b415 /tidb/cdc/default/default/upstream/7489358931240242907 {"id":7489358931240242907,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03adf792-3dd5-4b22-b4de-67c99245b415 {"id":"03adf792-3dd5-4b22-b4de-67c99245b415","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752272} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbe524f2 03adf792-3dd5-4b22-b4de-67c99245b415 /tidb/cdc/default/default/upstream/7489358931240242907 {"id":7489358931240242907,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x wait process cdc.test exit for 1-th time... + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null }' + echo '{ "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null }' { "upstream_id": 7489358661568166943, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 457114195242254337, "checkpoint_time": "2025-04-04 15:37:50.669", "error": null } ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114195242254337, '"checkpoint_time":' '"2025-04-04' '15:37:50.669",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489358661568166943, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 457114195242254337, '"checkpoint_time":' '"2025-04-04' '15:37:50.669",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6557.out cli changefeed remove -c changefeed-initialize-error wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 17-th time, retry later Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 457114196553236482 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 3-th time... table ddl_manager.finish_mark not exists for 52-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:37:56 GMT < Content-Length: 854 < Content-Type: text/plain; charset=utf-8 < { [854 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/18c20ca7-1dba-4e11-9ffe-2eae1c32c145 {"id":"18c20ca7-1dba-4e11-9ffe-2eae1c32c145","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752274} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbf3b3db 18c20ca7-1dba-4e11-9ffe-2eae1c32c145 /tidb/cdc/default/default/upstream/7489358947017772680 {"id":7489358947017772680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/18c20ca7-1dba-4e11-9ffe-2eae1c32c145 {"id":"18c20ca7-1dba-4e11-9ffe-2eae1c32c145","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752274} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbf3b3db 18c20ca7-1dba-4e11-9ffe-2eae1c32c145 /tidb/cdc/default/default/upstream/7489358947017772680 {"id":7489358947017772680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/18c20ca7-1dba-4e11-9ffe-2eae1c32c145 {"id":"18c20ca7-1dba-4e11-9ffe-2eae1c32c145","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752274} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbbf3b3db 18c20ca7-1dba-4e11-9ffe-2eae1c32c145 /tidb/cdc/default/default/upstream/7489358947017772680 {"id":7489358947017772680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: b8ba20f2-b169-45dc-b464-080a94eeb55b Info: {"upstream_id":7489358947017772680,"namespace":"default","id":"b8ba20f2-b169-45dc-b464-080a94eeb55b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:37:56.903193592+08:00","start_ts":457114196858109953,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114196858109953,"checkpoint_ts":457114196858109953,"checkpoint_time":"2025-04-04 15:37:56.833"} split_and_random_merge scale: 20 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit run test case test_owner_cleanup_stale_tasks [Fri Apr 4 15:37:57 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.35173519.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 *************************** 1. row *************************** count(distinct region_id): 1 [2025/04/04 15:37:57.117 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] table partition_table.t2 not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... + set +x Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358976227140674 TiDB Cluster ID. tikv_gc_leader_uuid 657fef0a0f40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:4371, start at 2025-04-04 15:37:56.031536381 +0800 CST m=+2.123616075 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:56.043 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:56.029 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:56.029 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358976227140674 TiDB Cluster ID. tikv_gc_leader_uuid 657fef0a0f40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:4371, start at 2025-04-04 15:37:56.031536381 +0800 CST m=+2.123616075 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:56.043 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:56.029 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:56.029 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489358969610301444 TiDB Cluster ID. tikv_gc_leader_uuid 657fef0a3200008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:4433, start at 2025-04-04 15:37:56.181239455 +0800 CST m=+2.208974218 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:39:56.190 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:37:56.168 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:27:56.168 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 18-th time, retry later wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 53-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_gbk Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... [2025/04/04 15:37:59.122 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:37:59 CST 2025] <<<<<< run test case changefeed_error success! >>>>>> table partition_table.t2 not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5405.out cli changefeed pause -c test + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:00 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039cb320 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114197397340165 MinTableBarrierTs:457114197659484165 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114197659484162, checkpointTs: 457114197659484162, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114197659484165, checkpointTs: 457114197659484165, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114197659484165, checkpointTs: 457114197659484165, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114197659484162,"min-table-barrier-ts":457114197659484165,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039cb320 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114197397340165 MinTableBarrierTs:457114197659484165 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114197659484162, checkpointTs: 457114197659484162, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114197659484165, checkpointTs: 457114197659484165, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114197659484165, checkpointTs: 457114197659484165, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114197659484162,"min-table-barrier-ts":457114197659484165,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0039cb320 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114197397340165 MinTableBarrierTs:457114197659484165 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114197659484162, checkpointTs: 457114197659484162, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114197659484165, checkpointTs: 457114197659484165, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114197659484165, checkpointTs: 457114197659484165, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114197659484162,"min-table-barrier-ts":457114197659484165,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table ddl_manager.finish_mark not exists for 54-th check, retry later [Fri Apr 4 15:38:00 CST 2025] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.58165818.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... check diff failed 19-th time, retry later [2025/04/04 15:38:01.127 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2025/04/04 15:38:01.132 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:01.134 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] found first match line: 457114197534179343 : [2025/04/04 15:38:00.117 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=134] [jobID=135] [query="CREATE TABLE IF NOT EXISTS `finishmark` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114197534179343] [2025/04/04 15:38:01.141 +08:00] [INFO] [case.go:503] ["all tables synced"] [endTs=457114197534179343] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table partition_table.t2 not exists for 6-th check, retry later ./t/alter_table.test: ok! 64 test cases passed, take time 17.932131109 s time="2025-04-04T15:38:01+08:00" level=info msg="run test [alter_table] ok" time="2025-04-04T15:38:01+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0000d2340 {{} {} 0} {0 0} [0xc000293480] {[]} 1 0xc00003ee70 false map[0xc000293480:map[0xc000293480:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" "is-owner": true, run task successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5433.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=false' --no-confirm Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=true To:mysql://root@127.0.0.1:3306/?batch-dml-enable=false} Update changefeed config successfully! ID: test Info: {"upstream_id":7489358918194230688,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=false","create_time":"2025-04-04T15:37:53.789880437+08:00","start_ts":457114196041793539,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":0,"checkpoint_ts":457114196841332742,"checkpoint_time":"2025-04-04 15:37:56.769"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff failed 20-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 55-th check, retry later owner pid: 3522 owner id 7f57e437-bebd-41a9-bc49-acd40e51e8ca [Fri Apr 4 15:38:03 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.36323634.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server2 --cluster-id default --addr 127.0.0.1:8301 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/052a89a2-4c6c-4f0a-a830-62b33e3f892c {"id":"052a89a2-4c6c-4f0a-a830-62b33e3f892c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752280} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc09e6d6 052a89a2-4c6c-4f0a-a830-62b33e3f892c /tidb/cdc/default/default/upstream/7489358976227140674 {"id":7489358976227140674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/052a89a2-4c6c-4f0a-a830-62b33e3f892c {"id":"052a89a2-4c6c-4f0a-a830-62b33e3f892c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752280} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc09e6d6 052a89a2-4c6c-4f0a-a830-62b33e3f892c /tidb/cdc/default/default/upstream/7489358976227140674 {"id":7489358976227140674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/052a89a2-4c6c-4f0a-a830-62b33e3f892c {"id":"052a89a2-4c6c-4f0a-a830-62b33e3f892c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752280} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc09e6d6 052a89a2-4c6c-4f0a-a830-62b33e3f892c /tidb/cdc/default/default/upstream/7489358976227140674 {"id":7489358976227140674,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later *************************** 1. row *************************** count(distinct region_id): 20 + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5463.out cli changefeed resume -c test [2025/04/04 15:38:03.960 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark1000] [2025/04/04 15:38:03.960 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark1000] PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 21-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 56-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 2-th check, retry later table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 not exists for 1-th check, retry later [2025/04/04 15:38:05.960 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark1000] + set +x [2025/04/04 15:38:06.037 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark1000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:06.039 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:06 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d2cab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114198970204169 MinTableBarrierTs:457114199232610313 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/capture/8ee021bf-fe29-4356-8674-f213cdeb843d {"id":"8ee021bf-fe29-4356-8674-f213cdeb843d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752283} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd85 8ee021bf-fe29-4356-8674-f213cdeb843d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114198970204169,"min-table-barrier-ts":457114199232610313,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8ee021bf-fe29-4356-8674-f213cdeb843d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d2cab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114198970204169 MinTableBarrierTs:457114199232610313 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/capture/8ee021bf-fe29-4356-8674-f213cdeb843d {"id":"8ee021bf-fe29-4356-8674-f213cdeb843d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752283} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd85 8ee021bf-fe29-4356-8674-f213cdeb843d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114198970204169,"min-table-barrier-ts":457114199232610313,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8ee021bf-fe29-4356-8674-f213cdeb843d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d2cab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114198970204169 MinTableBarrierTs:457114199232610313 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/capture/8ee021bf-fe29-4356-8674-f213cdeb843d {"id":"8ee021bf-fe29-4356-8674-f213cdeb843d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752283} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd85 8ee021bf-fe29-4356-8674-f213cdeb843d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114198970204169,"min-table-barrier-ts":457114199232610313,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8ee021bf-fe29-4356-8674-f213cdeb843d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "7f57e437-bebd-41a9-bc49-acd40e51e8ca" | grep id ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 57-th check, retry later table partition_table.finish_mark not exists for 3-th check, retry later check diff failed 22-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 not exists for 1-th check, retry later "cluster-id": "default" "id": "8ee021bf-fe29-4356-8674-f213cdeb843d", "cluster-id": "default" run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359016889209547 TiDB Cluster ID. tikv_gc_leader_uuid 657fef135f40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:4160, start at 2025-04-04 15:38:05.607435676 +0800 CST m=+2.311284016 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:05.615 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:05.660 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:05.660 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359016889209547 TiDB Cluster ID. tikv_gc_leader_uuid 657fef135f40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:4160, start at 2025-04-04 15:38:05.607435676 +0800 CST m=+2.311284016 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:05.615 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:05.660 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:05.660 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359017780000641 TiDB Cluster ID. tikv_gc_leader_uuid 657fef139140006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:4220, start at 2025-04-04 15:38:05.773807192 +0800 CST m=+2.393469097 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:05.781 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:05.765 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:05.765 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } split_and_random_merge scale: 40 *************************** 1. row *************************** count(distinct region_id): 6 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5515.out cli changefeed pause -c test PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:38:08.143 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark1000] [2025/04/04 15:38:08.149 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark1000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:08.151 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] found first match line: 457114199165763585 : [2025/04/04 15:38:06.167 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=136] [jobID=137] [query="CREATE TABLE IF NOT EXISTS `finishmark1000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114199165763585] [2025/04/04 15:38:08.243 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark1000] [2025/04/04 15:38:08.243 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114199165763585] [2025/04/04 15:38:08.246 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:08.250 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:08.255 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:08.260 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:08.738 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark2000] [2025/04/04 15:38:08.738 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark2000] table ddl_manager.finish_mark not exists for 58-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'val: 22' ____________________________________ *************************** 1. row *************************** id: 1 val: 1 *************************** 2. row *************************** id: 2 val: 2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:38:06 CST 2025] <<<<<< run test case ddl_puller_lag success! >>>>>> capture_id: 8ee021bf-fe29-4356-8674-f213cdeb843d check diff failed 23-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359021280007638 TiDB Cluster ID. tikv_gc_leader_uuid 657fef153b80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:4308, start at 2025-04-04 15:38:07.473163101 +0800 CST m=+2.558480653 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:07.481 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:07.470 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:07.470 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359021280007638 TiDB Cluster ID. tikv_gc_leader_uuid 657fef153b80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:4308, start at 2025-04-04 15:38:07.473163101 +0800 CST m=+2.558480653 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:07.481 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:07.470 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:07.470 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359013201840093 TiDB Cluster ID. tikv_gc_leader_uuid 657fef153c80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:4367, start at 2025-04-04 15:38:07.487670913 +0800 CST m=+2.496413061 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:07.496 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:07.474 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:07.474 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash-proxy.toml"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5546.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' --no-confirm =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Fri Apr 4 15:38:09 CST 2025] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.55265528.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.finish_mark not exists for 4-th check, retry later Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=false To:mysql://root@127.0.0.1:3306/?batch-dml-enable=true} Update changefeed config successfully! ID: test Info: {"upstream_id":7489358918194230688,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2025-04-04T15:37:53.789880437+08:00","start_ts":457114196041793539,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":0,"checkpoint_ts":457114196972404750,"checkpoint_time":"2025-04-04 15:37:57.269"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table changefeed_pause_resume.t3 exists ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 59-th check, retry later [2025/04/04 15:38:10.738 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] [2025/04/04 15:38:10.745 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark2000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:10.746 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5582.out cli changefeed resume -c test table partition_table.finish_mark not exists for 5-th check, retry later check diff failed 24-th time, retry later [Fri Apr 4 15:38:11 CST 2025] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.56185620.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... 0 [Fri Apr 4 15:38:12 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.37683770.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server3.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server3 --cluster-id default --addr 127.0.0.1:8302 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:12 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcc6c7b5-af6c-4bab-a67f-d42bade5cdba {"id":"bcc6c7b5-af6c-4bab-a67f-d42bade5cdba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752290} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc3088d6 bcc6c7b5-af6c-4bab-a67f-d42bade5cdba /tidb/cdc/default/default/upstream/7489359016889209547 {"id":7489359016889209547,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcc6c7b5-af6c-4bab-a67f-d42bade5cdba {"id":"bcc6c7b5-af6c-4bab-a67f-d42bade5cdba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752290} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc3088d6 bcc6c7b5-af6c-4bab-a67f-d42bade5cdba /tidb/cdc/default/default/upstream/7489359016889209547 {"id":7489359016889209547,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcc6c7b5-af6c-4bab-a67f-d42bade5cdba {"id":"bcc6c7b5-af6c-4bab-a67f-d42bade5cdba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752290} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc3088d6 bcc6c7b5-af6c-4bab-a67f-d42bade5cdba /tidb/cdc/default/default/upstream/7489359016889209547 {"id":7489359016889209547,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:38:12.841 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] [2025/04/04 15:38:12.936 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark2000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:12.938 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark3000] [2025/04/04 15:38:12.938 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark3000] [2025/04/04 15:38:12.938 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table ddl_manager.finish_mark not exists for 60-th check, retry later found first match line: 457114200424054792 : [2025/04/04 15:38:11.336 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=138] [jobID=139] [query="CREATE TABLE IF NOT EXISTS `finishmark2000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114200424054792] [2025/04/04 15:38:13.142 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark2000] [2025/04/04 15:38:13.142 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114200424054792] [2025/04/04 15:38:13.145 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:13.148 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:13.153 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:13.237 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + set +x table partition_table.finish_mark not exists for 6-th check, retry later check diff successfully table batch_update_to_no_batch.v1 exists table batch_update_to_no_batch.recover_and_insert not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:14 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9bd44edd-4f52-4de2-b30c-9faecf5f9a24 {"id":"9bd44edd-4f52-4de2-b30c-9faecf5f9a24","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752291} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc32ebd5 9bd44edd-4f52-4de2-b30c-9faecf5f9a24 /tidb/cdc/default/default/upstream/7489359021280007638 {"id":7489359021280007638,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9bd44edd-4f52-4de2-b30c-9faecf5f9a24 {"id":"9bd44edd-4f52-4de2-b30c-9faecf5f9a24","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752291} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc32ebd5 9bd44edd-4f52-4de2-b30c-9faecf5f9a24 /tidb/cdc/default/default/upstream/7489359021280007638 {"id":7489359021280007638,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9bd44edd-4f52-4de2-b30c-9faecf5f9a24 {"id":"9bd44edd-4f52-4de2-b30c-9faecf5f9a24","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752291} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc32ebd5 9bd44edd-4f52-4de2-b30c-9faecf5f9a24 /tidb/cdc/default/default/upstream/7489359021280007638 {"id":7489359021280007638,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x wait process cdc.test exit for 2-th time... *************************** 1. row *************************** count(distinct region_id): 40 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:38:15 CST 2025] <<<<<< run test case clustered_index success! >>>>>> table ddl_manager.finish_mark not exists for 61-th check, retry later [2025/04/04 15:38:14.938 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] [2025/04/04 15:38:14.947 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark3000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:14.948 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table partition_table.finish_mark not exists for 7-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:15 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ba240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114199756898313 MinTableBarrierTs:457114200019042316 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/capture/8ee021bf-fe29-4356-8674-f213cdeb843d {"id":"8ee021bf-fe29-4356-8674-f213cdeb843d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752283} /tidb/cdc/default/__cdc_meta__/capture/9948c472-c914-4732-bf11-782ae969bdf1 {"id":"9948c472-c914-4732-bf11-782ae969bdf1","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752292} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd85 8ee021bf-fe29-4356-8674-f213cdeb843d /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fde0c 9948c472-c914-4732-bf11-782ae969bdf1 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114199756898313,"min-table-barrier-ts":457114200019042316,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8ee021bf-fe29-4356-8674-f213cdeb843d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9948c472-c914-4732-bf11-782ae969bdf1/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ba240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114199756898313 MinTableBarrierTs:457114200019042316 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/capture/8ee021bf-fe29-4356-8674-f213cdeb843d {"id":"8ee021bf-fe29-4356-8674-f213cdeb843d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752283} /tidb/cdc/default/__cdc_meta__/capture/9948c472-c914-4732-bf11-782ae969bdf1 {"id":"9948c472-c914-4732-bf11-782ae969bdf1","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752292} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd85 8ee021bf-fe29-4356-8674-f213cdeb843d /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fde0c 9948c472-c914-4732-bf11-782ae969bdf1 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114199756898313,"min-table-barrier-ts":457114200019042316,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8ee021bf-fe29-4356-8674-f213cdeb843d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9948c472-c914-4732-bf11-782ae969bdf1/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0042ba240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114199756898313 MinTableBarrierTs:457114200019042316 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f57e437-bebd-41a9-bc49-acd40e51e8ca {"id":"7f57e437-bebd-41a9-bc49-acd40e51e8ca","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752277} /tidb/cdc/default/__cdc_meta__/capture/8ee021bf-fe29-4356-8674-f213cdeb843d {"id":"8ee021bf-fe29-4356-8674-f213cdeb843d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752283} /tidb/cdc/default/__cdc_meta__/capture/9948c472-c914-4732-bf11-782ae969bdf1 {"id":"9948c472-c914-4732-bf11-782ae969bdf1","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752292} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd38 7f57e437-bebd-41a9-bc49-acd40e51e8ca /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdd85 8ee021bf-fe29-4356-8674-f213cdeb843d /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fde0c 9948c472-c914-4732-bf11-782ae969bdf1 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114199756898313,"min-table-barrier-ts":457114200019042316,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f57e437-bebd-41a9-bc49-acd40e51e8ca/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8ee021bf-fe29-4356-8674-f213cdeb843d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9948c472-c914-4732-bf11-782ae969bdf1/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' run task failed 1-th time, retry later check diff failed 2-th time, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table consistent_replicate_gbk.GBKTABLE not exists for 1-th check, retry later table batch_update_to_no_batch.recover_and_insert not exists for 2-th check, retry later ./t/bigint.test: ok! 138 test cases passed, take time 12.843805182 s time="2025-04-04T15:38:15+08:00" level=info msg="run test [bigint] ok" time="2025-04-04T15:38:15+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0000d2950 {{} {} 0} {0 0} [0xc000457c80] {[]} 1 0xc0004ac540 false map[0xc000457c80:map[0xc000457c80:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 62-th check, retry later ./t/composite_index.test: ok! 12 test cases passed, take time 1.5425978630000001 s time="2025-04-04T15:38:17+08:00" level=info msg="run test [composite_index] ok" time="2025-04-04T15:38:17+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0001340e8 {{} {} 0} {0 0} [0xc0001ff680] {[]} 1 0xc0004ac310 false map[0xc0001ff680:map[0xc0001ff680:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" table partition_table.finish_mark not exists for 8-th check, retry later valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Fri Apr 4 15:38:15 CST 2025] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ true != \n\o ]] + set +x + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.53365338.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:38:17.151 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] [2025/04/04 15:38:17.236 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark3000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:17.238 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] found first match line: 457114201446416398 : [2025/04/04 15:38:15.336 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=140] [jobID=141] [query="CREATE TABLE IF NOT EXISTS `finishmark3000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114201446416398] [2025/04/04 15:38:17.446 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark3000] [2025/04/04 15:38:17.447 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114201446416398] [2025/04/04 15:38:17.449 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:17.452 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:17.457 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:17.538 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:17.838 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark4000] [2025/04/04 15:38:17.838 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark4000] run task failed 2-th time, retry later check diff failed 3-th time, retry later table batch_update_to_no_batch.recover_and_insert exists table batch_update_to_no_batch.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table consistent_replicate_gbk.GBKTABLE exists table consistent_replicate_gbk.check1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 63-th check, retry later ./t/concurrent_ddl.test: ok! 8 test cases passed, take time 1.02086326 s time="2025-04-04T15:38:18+08:00" level=info msg="run test [concurrent_ddl] ok" time="2025-04-04T15:38:18+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000134138 {{} {} 0} {0 0} [0xc0004b1400] {[]} 1 0xc0004ac770 false map[0xc0004b1400:map[0xc0004b1400:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" split_and_random_merge scale: 80 *************************** 1. row *************************** count(distinct region_id): 16 [2025/04/04 15:38:19.838 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] [2025/04/04 15:38:19.849 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark4000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:19.851 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table partition_table.finish_mark not exists for 9-th check, retry later check diff failed 4-th time, retry later table batch_update_to_no_batch.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_gbk.check1 exists check diff successfully table ddl_manager.finish_mark not exists for 64-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table partition_table.finish_mark exists check diff successfully start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table batch_update_to_no_batch.finish_mark not exists for 3-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:38:22 CST 2025] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.58005802.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359073219681702 TiDB Cluster ID. tikv_gc_leader_uuid 657fef21bc00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:7609, start at 2025-04-04 15:38:20.279714346 +0800 CST m=+2.638821449 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:20.287 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:20.272 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:20.272 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359073219681702 TiDB Cluster ID. tikv_gc_leader_uuid 657fef21bc00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:7609, start at 2025-04-04 15:38:20.279714346 +0800 CST m=+2.638821449 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:20.287 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:20.272 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:20.272 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359075670005866 TiDB Cluster ID. tikv_gc_leader_uuid 657fef21d64000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:7676, start at 2025-04-04 15:38:20.40396296 +0800 CST m=+2.686932391 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:20.412 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:20.377 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:20.377 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' [2025/04/04 15:38:21.845 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark5000] [2025/04/04 15:38:21.845 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark5000] [2025/04/04 15:38:22.146 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] [2025/04/04 15:38:22.153 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark4000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:22.154 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] wait process cdc.test exit for 2-th time... found first match line: 457114202730921999 : [2025/04/04 15:38:20.266 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=142] [jobID=143] [query="CREATE TABLE IF NOT EXISTS `finishmark4000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114202730921999] [2025/04/04 15:38:22.449 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark4000] [2025/04/04 15:38:22.449 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114202730921999] [2025/04/04 15:38:22.452 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:22.539 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:22.544 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:22.551 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:38:23 CST 2025] <<<<<< run test case partition_table success! >>>>>> table ddl_manager.finish_mark not exists for 65-th check, retry later Verifying downstream PD is started... "is-owner": true, run task successfully nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table batch_update_to_no_batch.finish_mark not exists for 4-th check, retry later [2025/04/04 15:38:23.846 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2025/04/04 15:38:23.852 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark5000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:23.854 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8893.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) *************************** 1. row *************************** count(distinct region_id): 79 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 66-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ca76660c-b90e-4657-a021-d66b68a8244e {UpstreamID:7489359021280007638 Namespace:default ID:ca76660c-b90e-4657-a021-d66b68a8244e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:14.628506458 +0800 CST StartTs:457114201494650882 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004348240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114201520865283} {CheckpointTs:457114203971911686 MinTableBarrierTs:457114203971911686 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114203971911686, checkpointTs: 457114203971911686, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114203971911686, checkpointTs: 457114203971911686, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3 {"id":"69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752302} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc32ec7d 69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3 /tidb/cdc/default/default/changefeed/info/ca76660c-b90e-4657-a021-d66b68a8244e {"upstream-id":7489359021280007638,"namespace":"default","changefeed-id":"ca76660c-b90e-4657-a021-d66b68a8244e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:38:14.628506458+08:00","start-ts":457114201494650882,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114201520865283} /tidb/cdc/default/default/changefeed/status/ca76660c-b90e-4657-a021-d66b68a8244e {"checkpoint-ts":457114203971911686,"min-table-barrier-ts":457114204234055688,"admin-job-type":0} /tidb/cdc/default/default/task/position/69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3/ca76660c-b90e-4657-a021-d66b68a8244e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359021280007638 {"id":7489359021280007638,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ca76660c-b90e-4657-a021-d66b68a8244e {UpstreamID:7489359021280007638 Namespace:default ID:ca76660c-b90e-4657-a021-d66b68a8244e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:14.628506458 +0800 CST StartTs:457114201494650882 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004348240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114201520865283} {CheckpointTs:457114203971911686 MinTableBarrierTs:457114203971911686 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114203971911686, checkpointTs: 457114203971911686, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114203971911686, checkpointTs: 457114203971911686, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3 {"id":"69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752302} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc32ec7d 69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3 /tidb/cdc/default/default/changefeed/info/ca76660c-b90e-4657-a021-d66b68a8244e {"upstream-id":7489359021280007638,"namespace":"default","changefeed-id":"ca76660c-b90e-4657-a021-d66b68a8244e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:38:14.628506458+08:00","start-ts":457114201494650882,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114201520865283} /tidb/cdc/default/default/changefeed/status/ca76660c-b90e-4657-a021-d66b68a8244e {"checkpoint-ts":457114203971911686,"min-table-barrier-ts":457114204234055688,"admin-job-type":0} /tidb/cdc/default/default/task/position/69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3/ca76660c-b90e-4657-a021-d66b68a8244e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359021280007638 {"id":7489359021280007638,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ca76660c-b90e-4657-a021-d66b68a8244e {UpstreamID:7489359021280007638 Namespace:default ID:ca76660c-b90e-4657-a021-d66b68a8244e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:14.628506458 +0800 CST StartTs:457114201494650882 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004348240 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114201520865283} {CheckpointTs:457114203971911686 MinTableBarrierTs:457114203971911686 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114203971911686, checkpointTs: 457114203971911686, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114203971911686, checkpointTs: 457114203971911686, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3 {"id":"69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752302} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc32ec7d 69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3 /tidb/cdc/default/default/changefeed/info/ca76660c-b90e-4657-a021-d66b68a8244e {"upstream-id":7489359021280007638,"namespace":"default","changefeed-id":"ca76660c-b90e-4657-a021-d66b68a8244e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:38:14.628506458+08:00","start-ts":457114201494650882,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114201520865283} /tidb/cdc/default/default/changefeed/status/ca76660c-b90e-4657-a021-d66b68a8244e {"checkpoint-ts":457114203971911686,"min-table-barrier-ts":457114204234055688,"admin-job-type":0} /tidb/cdc/default/default/task/position/69e7f5a5-8b8d-4fce-a03a-425bb2d2efa3/ca76660c-b90e-4657-a021-d66b68a8244e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359021280007638 {"id":7489359021280007638,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ./t/date_formats.test: ok! 99 test cases passed, take time 5.256262638 s time="2025-04-04T15:38:24+08:00" level=info msg="run test [date_formats] ok" time="2025-04-04T15:38:24+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000510340 {{} {} 0} {0 0} [0xc000666d80] {[]} 1 0xc0005311f0 false map[0xc000666d80:map[0xc000666d80:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" + set +x + tso='457114204116615169 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114204116615169 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:38:25 CST 2025] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.89298931.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_update_to_no_batch.finish_mark exists check diff successfully [2025/04/04 15:38:26.240 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2025/04/04 15:38:26.248 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark5000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:26.250 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] wait process cdc.test exit for 1-th time... found first match line: 457114203858141198 : [2025/04/04 15:38:24.366 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=144] [jobID=145] [query="CREATE TABLE IF NOT EXISTS `finishmark5000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114203858141198] [2025/04/04 15:38:26.537 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark5000] [2025/04/04 15:38:26.537 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114203858141198] [2025/04/04 15:38:26.540 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:26.544 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:26.549 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:26.641 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:26.655 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark6000] [2025/04/04 15:38:26.655 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark6000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 67-th check, retry later wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:38:27 CST 2025] <<<<<< run test case batch_update_to_no_batch success! >>>>>> nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later check diff failed 2-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 68-th check, retry later [2025/04/04 15:38:28.736 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] [2025/04/04 15:38:28.744 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark6000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:28.745 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:29 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a2d394e-e23d-4f00-9b19-f6b553f531e9 {"id":"4a2d394e-e23d-4f00-9b19-f6b553f531e9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752306} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc6a65e9 4a2d394e-e23d-4f00-9b19-f6b553f531e9 /tidb/cdc/default/default/upstream/7489359073219681702 {"id":7489359073219681702,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a2d394e-e23d-4f00-9b19-f6b553f531e9 {"id":"4a2d394e-e23d-4f00-9b19-f6b553f531e9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752306} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc6a65e9 4a2d394e-e23d-4f00-9b19-f6b553f531e9 /tidb/cdc/default/default/upstream/7489359073219681702 {"id":7489359073219681702,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a2d394e-e23d-4f00-9b19-f6b553f531e9 {"id":"4a2d394e-e23d-4f00-9b19-f6b553f531e9","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752306} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc6a65e9 4a2d394e-e23d-4f00-9b19-f6b553f531e9 /tidb/cdc/default/default/upstream/7489359073219681702 {"id":7489359073219681702,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8981.out cli changefeed create --start-ts=457114204116615169 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 81051182-a819-48c3-a67c-370987fc9c47 Info: {"upstream_id":7489359073219681702,"namespace":"default","id":"81051182-a819-48c3-a67c-370987fc9c47","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:38:29.491004221+08:00","start_ts":457114204116615169,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114204116615169,"checkpoint_ts":457114204116615169,"checkpoint_time":"2025-04-04 15:38:24.522"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) empty select id, val from test.availability1 where id=1 run task successfully test_owner_cleanup_stale_tasks pass table region_merge.t1 exists check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 3-th time, retry later wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 69-th check, retry later + set +x [2025/04/04 15:38:30.944 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark7000] [2025/04/04 15:38:30.944 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark7000] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_owner_retryable_error [Fri Apr 4 15:38:31 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/capture/capture-campaign-compacted-error=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.40044006.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:38:31.147 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] [2025/04/04 15:38:31.153 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark6000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:31.155 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359110825992106 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2aabc0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:6769, start at 2025-04-04 15:38:29.436451014 +0800 CST m=+2.162609472 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:29.444 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:29.423 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:29.423 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359110825992106 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2aabc0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:6769, start at 2025-04-04 15:38:29.436451014 +0800 CST m=+2.162609472 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:29.444 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:29.423 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:29.423 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359122104389400 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2ac4c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:6842, start at 2025-04-04 15:38:29.539162551 +0800 CST m=+2.181352137 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:29.546 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:29.523 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:29.523 +0800 All versions after safe point can be accessed. (DO NOT EDIT) found first match line: 457114205116432398 : [2025/04/04 15:38:29.366 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=146] [jobID=147] [query="CREATE TABLE IF NOT EXISTS `finishmark6000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114205116432398] [2025/04/04 15:38:31.546 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark6000] [2025/04/04 15:38:31.546 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114205116432398] [2025/04/04 15:38:31.554 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:31.642 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_sequence.finish_mark not exists for 1-th check, retry later [2025/04/04 15:38:31.737 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:31.746 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] check_redo_resolved_ts ca76660c-b90e-4657-a021-d66b68a8244e 457114205754490882 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/ca76660c-b90e-4657-a021-d66b68a8244e/meta global resolved ts 457114204273377303 not forward to 457114205754490882 run task failed 1-th time, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore check diff failed 4-th time, retry later Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359023120455598 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2c9b00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:3998, start at 2025-04-04 15:38:31.407180806 +0800 CST m=+27.082954049 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:31.415 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:31.404 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:31.404 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359023120455598 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2c9b00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:3998, start at 2025-04-04 15:38:31.407180806 +0800 CST m=+27.082954049 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:31.415 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:31.404 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:31.404 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359022179407967 TiDB Cluster ID. tikv_gc_leader_uuid 657fef13d7c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:4057, start at 2025-04-04 15:38:06.074429669 +0800 CST m=+1.694065196 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:06.082 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:06.047 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:06.047 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:38:32.944 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] [2025/04/04 15:38:32.952 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark7000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:32.954 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] check_redo_resolved_ts ca76660c-b90e-4657-a021-d66b68a8244e 457114205754490882 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/ca76660c-b90e-4657-a021-d66b68a8244e/meta run task successfully table ddl_manager.finish_mark not exists for 70-th check, retry later found first match line: 457114206165008399 : [2025/04/04 15:38:33.366 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=148] [jobID=149] [query="CREATE TABLE IF NOT EXISTS `finishmark7000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114206165008399] [2025/04/04 15:38:33.446 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark7000] [2025/04/04 15:38:33.447 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114206165008399] [2025/04/04 15:38:33.449 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:33.452 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:33.456 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:33.461 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] wait process cdc.test exit for 1-th time... ./t/drop.test: ok! 48 test cases passed, take time 8.665700565 s time="2025-04-04T15:38:33+08:00" level=info msg="run test [drop] ok" time="2025-04-04T15:38:33+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0000d2260 {{} {} 0} {0 0} [0xc000666280] {[]} 1 0xc000530930 false map[0xc000666280:map[0xc000666280:true]] map[] 0 0 0 0 0 0 0 0 0x4a1c40}" table ddl_sequence.finish_mark not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... [Fri Apr 4 15:38:33 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.80928094.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:34 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004b40ab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114206585749509 MinTableBarrierTs:457114206585749512 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114206585749509, checkpointTs: 457114206585749509, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114206585749512, checkpointTs: 457114206585749512, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114206585749512, checkpointTs: 457114206585749512, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3b3a26a3-4169-422d-8b48-8de85262e9a4 {"id":"3b3a26a3-4169-422d-8b48-8de85262e9a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752311} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdece 3b3a26a3-4169-422d-8b48-8de85262e9a4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114206585749509,"min-table-barrier-ts":457114206585749512,"admin-job-type":0} /tidb/cdc/default/default/task/position/3b3a26a3-4169-422d-8b48-8de85262e9a4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004b40ab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114206585749509 MinTableBarrierTs:457114206585749512 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114206585749509, checkpointTs: 457114206585749509, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114206585749512, checkpointTs: 457114206585749512, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114206585749512, checkpointTs: 457114206585749512, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3b3a26a3-4169-422d-8b48-8de85262e9a4 {"id":"3b3a26a3-4169-422d-8b48-8de85262e9a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752311} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdece 3b3a26a3-4169-422d-8b48-8de85262e9a4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114206585749509,"min-table-barrier-ts":457114206585749512,"admin-job-type":0} /tidb/cdc/default/default/task/position/3b3a26a3-4169-422d-8b48-8de85262e9a4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004b40ab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114206585749509 MinTableBarrierTs:457114206585749512 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114206585749509, checkpointTs: 457114206585749509, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114206585749512, checkpointTs: 457114206585749512, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114206585749512, checkpointTs: 457114206585749512, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3b3a26a3-4169-422d-8b48-8de85262e9a4 {"id":"3b3a26a3-4169-422d-8b48-8de85262e9a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752311} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdece 3b3a26a3-4169-422d-8b48-8de85262e9a4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114206585749509,"min-table-barrier-ts":457114206585749512,"admin-job-type":0} /tidb/cdc/default/default/task/position/3b3a26a3-4169-422d-8b48-8de85262e9a4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [2025/04/04 15:38:34.602 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:38:34.603 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:38:34.603 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:38:34.604 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:38:34.607 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:38:34.607 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114204273377303] [resolvedTs=457114206069063687] [2025/04/04 15:38:34.607 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:38:34.608 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:38:34.608 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:38:34.608 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:38:34.608 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:38:34.611 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=4.336779ms] [2025/04/04 15:38:34.611 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=4.904349ms] [2025/04/04 15:38:34.613 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:38:34.616 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:38:34.618 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:38:34.618 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:38:34.618 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:38:34.618 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:38:34.623 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:38:34.626 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:38:34.626 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:38:34.626 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114204273377299,\"CommitTs\":457114204273377303,\"Query\":\"create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_gbk\",\"Table\":\"GBKTABLE2\",\"TableID\":118,\"IsPartition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:38:34.626 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114204273377303] [DDL="create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE"] [2025/04/04 15:38:34.628 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114204273377299] [ddl="create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE"] [namespace=] [changefeed=] [error="Error 1050 (42S01): Table 'consistent_replicate_gbk.GBKTABLE2' already exists"] [Fri Apr 4 15:38:34 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.54055407.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:38:34.829 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114206069063687] [2025/04/04 15:38:34.829 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114206069063687] [2025/04/04 15:38:34.929 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=1] [appliedDDLCount=1] [currentCheckpoint=457114206069063687] [2025/04/04 15:38:34.929 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:38:34.929 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:38:34.929 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:38:34.930 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] Apply redo log successfully wait process cdc.test exit for 1-th time... check diff successfully [Fri Apr 4 15:38:35 CST 2025] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT table ddl_manager.finish_mark not exists for 71-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_with_exists/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:38:33 CST 2025] <<<<<< run test case ddl_with_exists success! >>>>>> wait process cdc.test exit for 2-th time... invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Fri Apr 4 15:38:35 CST 2025] <<<<<< run all test cases cdc_server_tips success! >>>>>> ./t/mysql_replace.test: ok! 16 test cases passed, take time 1.0460282890000001 s time="2025-04-04T15:38:35+08:00" level=info msg="run test [mysql_replace] ok" Great, All tests passed mysqltest end table ddl_sequence.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 1-th check, retry later "is-owner": true, run task successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:38:35 CST 2025] <<<<<< run test case region_merge success! >>>>>> ***************** properties ***************** "workload"="core" "requestdistribution"="uniform" "mysql.db"="testSync" "operationcount"="0" "readproportion"="0" "scanproportion"="0" "mysql.user"="root" "readallfields"="true" "mysql.host"="127.0.0.1" "updateproportion"="0" "dotransactions"="false" "mysql.port"="4000" "threadcount"="2" "recordcount"="10" "insertproportion"="0" ********************************************** Run finished, takes 4.50972ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3377.6, Avg(us): 808, Min(us): 437, Max(us): 2074, 95th(us): 3000, 99th(us): 3000 [2025/04/04 15:38:35.547 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark8000] [2025/04/04 15:38:35.547 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark8000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:36 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de840fc-50ef-46ad-a634-91e98427a6d8 {"id":"0de840fc-50ef-46ad-a634-91e98427a6d8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c26d7 0de840fc-50ef-46ad-a634-91e98427a6d8 /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de840fc-50ef-46ad-a634-91e98427a6d8 {"id":"0de840fc-50ef-46ad-a634-91e98427a6d8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c26d7 0de840fc-50ef-46ad-a634-91e98427a6d8 /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0de840fc-50ef-46ad-a634-91e98427a6d8 {"id":"0de840fc-50ef-46ad-a634-91e98427a6d8","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752314} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c26d7 0de840fc-50ef-46ad-a634-91e98427a6d8 /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.8162.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-only-block-related-table VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359136756082465 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2fb20000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6, pid:4551, start at 2025-04-04 15:38:34.604784798 +0800 CST m=+2.130984366 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:34.612 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:34.617 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:34.617 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359136756082465 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2fb20000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6, pid:4551, start at 2025-04-04 15:38:34.604784798 +0800 CST m=+2.130984366 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:34.612 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:34.617 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:34.617 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359140353739515 TiDB Cluster ID. tikv_gc_leader_uuid 657fef2fd640004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zpm8c-13z7t-zjcz6, pid:4609, start at 2025-04-04 15:38:34.717273269 +0800 CST m=+2.186439163 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:34.725 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:34.762 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:34.762 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 72-th check, retry later Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7489359110825992106,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:38:37.223216981+08:00","start_ts":457114207417270275,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114207417270275,"checkpoint_ts":457114207417270275,"checkpoint_time":"2025-04-04 15:38:37.113"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_resume_with_checkpoint_ts/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... owner pid: 4009 owner id 3b3a26a3-4169-422d-8b48-8de85262e9a4 [Fri Apr 4 15:38:37 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/owner-run-with-error=1*return(true);github.com/pingcap/tiflow/cdc/capture/capture-resign-failed=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.41134115.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:38:37.547 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] [2025/04/04 15:38:37.553 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark8000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:37.555 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table ddl_sequence.finish_mark not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:37 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d15a11b2-b191-4401-9655-0b373efa5df6 {"id":"d15a11b2-b191-4401-9655-0b373efa5df6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752315} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc388638 d15a11b2-b191-4401-9655-0b373efa5df6 /tidb/cdc/default/default/upstream/7489359023120455598 {"id":7489359023120455598,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d15a11b2-b191-4401-9655-0b373efa5df6 {"id":"d15a11b2-b191-4401-9655-0b373efa5df6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752315} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc388638 d15a11b2-b191-4401-9655-0b373efa5df6 /tidb/cdc/default/default/upstream/7489359023120455598 {"id":7489359023120455598,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d15a11b2-b191-4401-9655-0b373efa5df6 {"id":"d15a11b2-b191-4401-9655-0b373efa5df6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752315} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc388638 d15a11b2-b191-4401-9655-0b373efa5df6 /tidb/cdc/default/default/upstream/7489359023120455598 {"id":7489359023120455598,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.5450.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 3469888e-572c-46cf-9af5-3c22a1573568 Info: {"upstream_id":7489359023120455598,"namespace":"default","id":"3469888e-572c-46cf-9af5-3c22a1573568","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:38:38.251141875+08:00","start_ts":457114207690162182,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114207690162182,"checkpoint_ts":457114207690162182,"checkpoint_time":"2025-04-04 15:38:38.154"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... \033[0;36m<<< Run all test success >>>\033[0m check diff failed 1-th time, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) + set +x [Pipeline] // cache [Pipeline] } table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later [Pipeline] // dir [Fri Apr 4 15:38:38 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.58945896.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 73-th check, retry later [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [2025/04/04 15:38:39.564 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark9000] [2025/04/04 15:38:39.564 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark9000] + set +x [Pipeline] // withEnv [Pipeline] } [2025/04/04 15:38:39.854 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] [2025/04/04 15:38:39.944 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark8000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:39.946 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table ddl_sequence.finish_mark not exists for 5-th check, retry later [Pipeline] // node [Pipeline] } table test.finish_mark not exists for 3-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } check diff failed 2-th time, retry later [Pipeline] // stage [Pipeline] } found first match line: 457114207501942798 : [2025/04/04 15:38:38.436 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=150] [jobID=151] [query="CREATE TABLE IF NOT EXISTS `finishmark8000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114207501942798] [2025/04/04 15:38:40.542 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark8000] [2025/04/04 15:38:40.542 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114207501942798] [2025/04/04 15:38:40.544 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:40.548 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:40.552 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:40.649 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 74-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:40 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b89d40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114208158351368 MinTableBarrierTs:457114208158351370 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3b3a26a3-4169-422d-8b48-8de85262e9a4 {"id":"3b3a26a3-4169-422d-8b48-8de85262e9a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752311} /tidb/cdc/default/__cdc_meta__/capture/8e3808e4-c32b-428b-8bb3-1206788cb38d {"id":"8e3808e4-c32b-428b-8bb3-1206788cb38d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752317} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdece 3b3a26a3-4169-422d-8b48-8de85262e9a4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf29 8e3808e4-c32b-428b-8bb3-1206788cb38d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114208158351368,"min-table-barrier-ts":457114208158351370,"admin-job-type":0} /tidb/cdc/default/default/task/position/3b3a26a3-4169-422d-8b48-8de85262e9a4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8e3808e4-c32b-428b-8bb3-1206788cb38d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b89d40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114208158351368 MinTableBarrierTs:457114208158351370 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3b3a26a3-4169-422d-8b48-8de85262e9a4 {"id":"3b3a26a3-4169-422d-8b48-8de85262e9a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752311} /tidb/cdc/default/__cdc_meta__/capture/8e3808e4-c32b-428b-8bb3-1206788cb38d {"id":"8e3808e4-c32b-428b-8bb3-1206788cb38d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752317} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdece 3b3a26a3-4169-422d-8b48-8de85262e9a4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf29 8e3808e4-c32b-428b-8bb3-1206788cb38d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114208158351368,"min-table-barrier-ts":457114208158351370,"admin-job-type":0} /tidb/cdc/default/default/task/position/3b3a26a3-4169-422d-8b48-8de85262e9a4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8e3808e4-c32b-428b-8bb3-1206788cb38d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003b89d40 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114208158351368 MinTableBarrierTs:457114208158351370 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3b3a26a3-4169-422d-8b48-8de85262e9a4 {"id":"3b3a26a3-4169-422d-8b48-8de85262e9a4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752311} /tidb/cdc/default/__cdc_meta__/capture/8e3808e4-c32b-428b-8bb3-1206788cb38d {"id":"8e3808e4-c32b-428b-8bb3-1206788cb38d","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752317} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdece 3b3a26a3-4169-422d-8b48-8de85262e9a4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf29 8e3808e4-c32b-428b-8bb3-1206788cb38d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114208158351368,"min-table-barrier-ts":457114208158351370,"admin-job-type":0} /tidb/cdc/default/default/task/position/3b3a26a3-4169-422d-8b48-8de85262e9a4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8e3808e4-c32b-428b-8bb3-1206788cb38d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "3b3a26a3-4169-422d-8b48-8de85262e9a4" | grep id [2025/04/04 15:38:41.564 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] table ddl_sequence.finish_mark not exists for 6-th check, retry later "cluster-id": "default" "id": "8e3808e4-c32b-428b-8bb3-1206788cb38d", "cluster-id": "default" run task successfully table test.finish_mark not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:41 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72e61136-1eef-4a57-aa8e-76aa52346793 {"id":"72e61136-1eef-4a57-aa8e-76aa52346793","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752319} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca464d8 72e61136-1eef-4a57-aa8e-76aa52346793 /tidb/cdc/default/default/upstream/7489359136756082465 {"id":7489359136756082465,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72e61136-1eef-4a57-aa8e-76aa52346793 {"id":"72e61136-1eef-4a57-aa8e-76aa52346793","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752319} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca464d8 72e61136-1eef-4a57-aa8e-76aa52346793 /tidb/cdc/default/default/upstream/7489359136756082465 {"id":7489359136756082465,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72e61136-1eef-4a57-aa8e-76aa52346793 {"id":"72e61136-1eef-4a57-aa8e-76aa52346793","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752319} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca464d8 72e61136-1eef-4a57-aa8e-76aa52346793 /tidb/cdc/default/default/upstream/7489359136756082465 {"id":7489359136756082465,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.5954.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 Create changefeed successfully! ID: 22e6553d-2af6-4c37-baab-8ecb2ac2c92d Info: {"upstream_id":7489359136756082465,"namespace":"default","id":"22e6553d-2af6-4c37-baab-8ecb2ac2c92d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:38:42.347965356+08:00","start_ts":457114208768360453,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114208768360453,"checkpoint_ts":457114208768360453,"checkpoint_time":"2025-04-04 15:38:42.267"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 75-th check, retry later check diff failed 3-th time, retry later table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:38:43.638 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] table ddl_sequence.finish_mark not exists for 7-th check, retry later [2025/04/04 15:38:43.646 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark9000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:43.649 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] + set +x capture_id: 8e3808e4-c32b-428b-8bb3-1206788cb38d % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 35 100 35 0 0 35000 0 --:--:-- --:--:-- --:--:-- 35000 { "status": true, "message": "" }ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '3b3a26a3-4169-422d-8b48-8de85262e9a4' 3b3a26a3-4169-422d-8b48-8de85262e9a4 run task successfully ps -C cdc.test -o pid= | awk '{print $1}' | wc -l | grep 2 2 run task successfully test_owner_retryable_error pass table test.finish_mark not exists for 5-th check, retry later wait process cdc.test exit for 1-th time... found first match line: 457114208602947598 : [2025/04/04 15:38:42.541 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=152] [jobID=153] [query="CREATE TABLE IF NOT EXISTS `finishmark9000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114208602947598] [2025/04/04 15:38:44.346 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark9000] [2025/04/04 15:38:44.347 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114208602947598] [2025/04/04 15:38:44.350 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:44.354 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:44.438 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:44.444 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:44.445 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark10000] [2025/04/04 15:38:44.445 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark10000] wait process cdc.test exit for 2-th time... check diff failed 4-th time, retry later table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 exists table processor_delay.t4 exists table processor_delay.t5 exists table processor_delay.t6 exists table processor_delay.t7 exists table processor_delay.t8 exists table processor_delay.t9 exists table processor_delay.t10 exists table processor_delay.t11 exists table processor_delay.t12 exists table processor_delay.t13 not exists for 1-th check, retry later table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 exists [Fri Apr 4 15:38:45 CST 2025] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.60306032.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_gap_between_watch_capture [Fri Apr 4 15:38:45 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.42754277.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 76-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t14 exists table processor_delay.t15 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 8-th check, retry later table test.finish_mark not exists for 6-th check, retry later check diff successfully [2025/04/04 15:38:46.445 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] table ddl_only_block_related_table.finish_mark exists table ddl_manager.finish_mark not exists for 77-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 8097 exit for 1-th time... table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later table ddl_sequence.finish_mark exists table test.finish_mark not exists for 7-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff successfully wait process 8097 exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:38:48.144 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark11000] [2025/04/04 15:38:48.144 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark11000] start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_nfs Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 1-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8097) - No such process wait process 8097 exit for 3-th time... process 8097 already exit [Fri Apr 4 15:38:48 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.82518253.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:48 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {UpstreamID:7489359136756082465 Namespace:default ID:22e6553d-2af6-4c37-baab-8ecb2ac2c92d SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:42.347965356 +0800 CST StartTs:457114208768360453 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00441a5a0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114208781729798} {CheckpointTs:457114210144616452 MinTableBarrierTs:457114210144616452 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72e61136-1eef-4a57-aa8e-76aa52346793 {"id":"72e61136-1eef-4a57-aa8e-76aa52346793","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752319} /tidb/cdc/default/__cdc_meta__/capture/ec6b2156-5a2d-4917-8605-ce3d306fd533 {"id":"ec6b2156-5a2d-4917-8605-ce3d306fd533","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca464d8 72e61136-1eef-4a57-aa8e-76aa52346793 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca46550 ec6b2156-5a2d-4917-8605-ce3d306fd533 /tidb/cdc/default/default/changefeed/info/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"upstream-id":7489359136756082465,"namespace":"default","changefeed-id":"22e6553d-2af6-4c37-baab-8ecb2ac2c92d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:38:42.347965356+08:00","start-ts":457114208768360453,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114208781729798} /tidb/cdc/default/default/changefeed/status/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":457114210144616452,"min-table-barrier-ts":457114210144616452,"admin-job-type":0} /tidb/cdc/default/default/task/position/72e61136-1eef-4a57-aa8e-76aa52346793/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ec6b2156-5a2d-4917-8605-ce3d306fd533/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359136756082465 {"id":7489359136756082465,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {UpstreamID:7489359136756082465 Namespace:default ID:22e6553d-2af6-4c37-baab-8ecb2ac2c92d SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:42.347965356 +0800 CST StartTs:457114208768360453 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00441a5a0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114208781729798} {CheckpointTs:457114210144616452 MinTableBarrierTs:457114210144616452 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72e61136-1eef-4a57-aa8e-76aa52346793 {"id":"72e61136-1eef-4a57-aa8e-76aa52346793","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752319} /tidb/cdc/default/__cdc_meta__/capture/ec6b2156-5a2d-4917-8605-ce3d306fd533 {"id":"ec6b2156-5a2d-4917-8605-ce3d306fd533","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca464d8 72e61136-1eef-4a57-aa8e-76aa52346793 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca46550 ec6b2156-5a2d-4917-8605-ce3d306fd533 /tidb/cdc/default/default/changefeed/info/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"upstream-id":7489359136756082465,"namespace":"default","changefeed-id":"22e6553d-2af6-4c37-baab-8ecb2ac2c92d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:38:42.347965356+08:00","start-ts":457114208768360453,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114208781729798} /tidb/cdc/default/default/changefeed/status/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":457114210144616452,"min-table-barrier-ts":457114210144616452,"admin-job-type":0} /tidb/cdc/default/default/task/position/72e61136-1eef-4a57-aa8e-76aa52346793/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ec6b2156-5a2d-4917-8605-ce3d306fd533/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359136756082465 {"id":7489359136756082465,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {UpstreamID:7489359136756082465 Namespace:default ID:22e6553d-2af6-4c37-baab-8ecb2ac2c92d SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:42.347965356 +0800 CST StartTs:457114208768360453 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00441a5a0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114208781729798} {CheckpointTs:457114210144616452 MinTableBarrierTs:457114210144616452 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/72e61136-1eef-4a57-aa8e-76aa52346793 {"id":"72e61136-1eef-4a57-aa8e-76aa52346793","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752319} /tidb/cdc/default/__cdc_meta__/capture/ec6b2156-5a2d-4917-8605-ce3d306fd533 {"id":"ec6b2156-5a2d-4917-8605-ce3d306fd533","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca464d8 72e61136-1eef-4a57-aa8e-76aa52346793 /tidb/cdc/default/__cdc_meta__/owner/223195ffbca46550 ec6b2156-5a2d-4917-8605-ce3d306fd533 /tidb/cdc/default/default/changefeed/info/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"upstream-id":7489359136756082465,"namespace":"default","changefeed-id":"22e6553d-2af6-4c37-baab-8ecb2ac2c92d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:38:42.347965356+08:00","start-ts":457114208768360453,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114208781729798} /tidb/cdc/default/default/changefeed/status/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":457114210144616452,"min-table-barrier-ts":457114210144616452,"admin-job-type":0} /tidb/cdc/default/default/task/position/72e61136-1eef-4a57-aa8e-76aa52346793/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ec6b2156-5a2d-4917-8605-ce3d306fd533/22e6553d-2af6-4c37-baab-8ecb2ac2c92d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359136756082465 {"id":7489359136756082465,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:38:48.452 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] [2025/04/04 15:38:48.459 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark10000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:48.460 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] wait process cdc.test exit for 2-th time... found first match line: 457114209887453199 : [2025/04/04 15:38:47.536 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=154] [jobID=155] [query="CREATE TABLE IF NOT EXISTS `finishmark10000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114209887453199] [2025/04/04 15:38:49.056 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark10000] [2025/04/04 15:38:49.056 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114209887453199] table ddl_manager.finish_mark not exists for 78-th check, retry later [2025/04/04 15:38:49.139 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:49.144 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:49.148 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:49.153 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... table batch_add_table.finish_mark not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359191455856322 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3c48c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:6692, start at 2025-04-04 15:38:47.487547006 +0800 CST m=+2.440519693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:47.495 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:47.459 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:47.459 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359191455856322 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3c48c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:6692, start at 2025-04-04 15:38:47.487547006 +0800 CST m=+2.440519693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:47.495 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:47.459 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:47.459 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359195296915772 TiDB Cluster ID. tikv_gc_leader_uuid 657fef3c7500002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-vsll5-3qk5x-zm6sw, pid:6767, start at 2025-04-04 15:38:47.638259808 +0800 CST m=+2.472810160 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:47.646 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:47.636 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:47.636 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t16 exists table processor_delay.t17 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:38:49 CST 2025] <<<<<< run test case ddl_sequence success! >>>>>> table test.finish_mark not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... [2025/04/04 15:38:50.144 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2025/04/04 15:38:50.150 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark11000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:50.152 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... found first match line: 457114210778742798 : [2025/04/04 15:38:50.537 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=156] [jobID=157] [query="CREATE TABLE IF NOT EXISTS `finishmark11000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114210778742798] [2025/04/04 15:38:50.846 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark11000] [2025/04/04 15:38:50.847 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114210778742798] check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table ddl_manager.finish_mark not exists for 79-th check, retry later table batch_add_table.finish_mark not exists for 2-th check, retry later [2025/04/04 15:38:50.939 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:50.944 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:51.037 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:51.044 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] start tidb cluster in /tmp/tidb_cdc_test/common_1 Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully killing cdc1: 5899 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later [Fri Apr 4 15:38:51 CST 2025] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.79967998.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489359110825992106 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:37.223216981 +0800 CST StartTs:457114207417270275 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004449560 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114207443484674} {CheckpointTs:457114210877571078 MinTableBarrierTs:457114211152822273 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d0b1ff4-6511-4bf1-9181-c60befe360cf {"id":"0d0b1ff4-6511-4bf1-9181-c60befe360cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752328} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c27a2 0d0b1ff4-6511-4bf1-9181-c60befe360cf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489359110825992106,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:38:37.223216981+08:00","start-ts":457114207417270275,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114207443484674} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114210877571078,"min-table-barrier-ts":457114211152822273,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d0b1ff4-6511-4bf1-9181-c60befe360cf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489359110825992106 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:37.223216981 +0800 CST StartTs:457114207417270275 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004449560 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114207443484674} {CheckpointTs:457114210877571078 MinTableBarrierTs:457114211152822273 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d0b1ff4-6511-4bf1-9181-c60befe360cf {"id":"0d0b1ff4-6511-4bf1-9181-c60befe360cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752328} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c27a2 0d0b1ff4-6511-4bf1-9181-c60befe360cf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489359110825992106,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:38:37.223216981+08:00","start-ts":457114207417270275,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114207443484674} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114210877571078,"min-table-barrier-ts":457114211152822273,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d0b1ff4-6511-4bf1-9181-c60befe360cf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489359110825992106 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:37.223216981 +0800 CST StartTs:457114207417270275 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004449560 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114207443484674} {CheckpointTs:457114210877571078 MinTableBarrierTs:457114211152822273 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114211152822273, checkpointTs: 457114211152822273, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d0b1ff4-6511-4bf1-9181-c60befe360cf {"id":"0d0b1ff4-6511-4bf1-9181-c60befe360cf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752328} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c27a2 0d0b1ff4-6511-4bf1-9181-c60befe360cf /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489359110825992106,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:38:37.223216981+08:00","start-ts":457114207417270275,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114207443484674} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114210877571078,"min-table-barrier-ts":457114211152822273,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d0b1ff4-6511-4bf1-9181-c60befe360cf/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later check diff failed 1-th time, retry later table test.finish_mark not exists for 9-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:38:40.624 +08:00] [WARN] [diff.go:180] ["table struct is not equal"] [reason="column num not equal, one is 34 another is 11"] [2025/04/04 15:38:45.626 +08:00] [WARN] [diff.go:180] ["table struct is not equal"] [reason="column num not equal, one is 34 another is 24"] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3932 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0049f7170 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114208945045514 MinTableBarrierTs:457114208945045516 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8338181a-3063-4dac-8b2e-1c8376bddd21 {"id":"8338181a-3063-4dac-8b2e-1c8376bddd21","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf9e 8338181a-3063-4dac-8b2e-1c8376bddd21 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114208945045514,"min-table-barrier-ts":457114208945045516,"admin-job-type":0} /tidb/cdc/default/default/task/position/8338181a-3063-4dac-8b2e-1c8376bddd21/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0049f7170 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114208945045514 MinTableBarrierTs:457114208945045516 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8338181a-3063-4dac-8b2e-1c8376bddd21 {"id":"8338181a-3063-4dac-8b2e-1c8376bddd21","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf9e 8338181a-3063-4dac-8b2e-1c8376bddd21 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114208945045514,"min-table-barrier-ts":457114208945045516,"admin-job-type":0} /tidb/cdc/default/default/task/position/8338181a-3063-4dac-8b2e-1c8376bddd21/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0049f7170 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114208945045514 MinTableBarrierTs:457114208945045516 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8338181a-3063-4dac-8b2e-1c8376bddd21 {"id":"8338181a-3063-4dac-8b2e-1c8376bddd21","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf9e 8338181a-3063-4dac-8b2e-1c8376bddd21 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114208945045514,"min-table-barrier-ts":457114208945045516,"admin-job-type":0} /tidb/cdc/default/default/task/position/8338181a-3063-4dac-8b2e-1c8376bddd21/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 80-th check, retry later table batch_add_table.finish_mark not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:38:53.342 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark12000] [2025/04/04 15:38:53.343 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark12000] table processor_delay.t18 exists table processor_delay.t19 not exists for 1-th check, retry later table test.finish_mark not exists for 10-th check, retry later "is-owner": true, run task successfully cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later check diff successfully check_ts_not_forward ddl-only-block-related-table Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:54 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/984ba730-20df-4a06-9de7-e54289da231f {"id":"984ba730-20df-4a06-9de7-e54289da231f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752331} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcd788d6 984ba730-20df-4a06-9de7-e54289da231f /tidb/cdc/default/default/upstream/7489359191455856322 {"id":7489359191455856322,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/984ba730-20df-4a06-9de7-e54289da231f {"id":"984ba730-20df-4a06-9de7-e54289da231f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752331} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcd788d6 984ba730-20df-4a06-9de7-e54289da231f /tidb/cdc/default/default/upstream/7489359191455856322 {"id":7489359191455856322,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/984ba730-20df-4a06-9de7-e54289da231f {"id":"984ba730-20df-4a06-9de7-e54289da231f","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752331} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcd788d6 984ba730-20df-4a06-9de7-e54289da231f /tidb/cdc/default/default/upstream/7489359191455856322 {"id":7489359191455856322,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table test.table1 not exists for 1-th check, retry later check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) owner pid: 4280 owner id 8338181a-3063-4dac-8b2e-1c8376bddd21 [Fri Apr 4 15:38:55 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.43974399.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 81-th check, retry later table batch_add_table.finish_mark not exists for 4-th check, retry later [2025/04/04 15:38:55.343 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] [2025/04/04 15:38:55.351 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark12000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:38:55.352 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later table test.finish_mark not exists for 11-th check, retry later found first match line: 457114211984605197 : [2025/04/04 15:38:55.536 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=158] [jobID=159] [query="CREATE TABLE IF NOT EXISTS `finishmark12000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114211984605197] [2025/04/04 15:38:56.049 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark12000] [2025/04/04 15:38:56.049 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114211984605197] [2025/04/04 15:38:56.052 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:56.055 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:38:56.059 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:38:56.064 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 4-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359229107577433 TiDB Cluster ID. tikv_gc_leader_uuid 657fef44308000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:6980, start at 2025-04-04 15:38:55.582789881 +0800 CST m=+2.654332519 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:55.591 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:55.554 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:55.554 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359229107577433 TiDB Cluster ID. tikv_gc_leader_uuid 657fef44308000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:6980, start at 2025-04-04 15:38:55.582789881 +0800 CST m=+2.654332519 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:55.591 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:55.554 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:55.554 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359233100042633 TiDB Cluster ID. tikv_gc_leader_uuid 657fef443c40009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:7052, start at 2025-04-04 15:38:55.614936664 +0800 CST m=+2.605382217 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:55.660 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:55.657 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:55.657 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table ddl_manager.finish_mark not exists for 82-th check, retry later table batch_add_table.finish_mark not exists for 5-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t20 exists table processor_delay.t21 not exists for 1-th check, retry later table test.table1 exists table test.table2 not exists for 1-th check, retry later check_changefeed_state http://127.0.0.1:2379 97dc0d50-bbfb-4590-a258-d60b5457f188 finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=97dc0d50-bbfb-4590-a258-d60b5457f188 + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 97dc0d50-bbfb-4590-a258-d60b5457f188 -s + info='{ "upstream_id": 7489358765827600768, "namespace": "default", "id": "97dc0d50-bbfb-4590-a258-d60b5457f188", "state": "finished", "checkpoint_tso": 457114209827422213, "checkpoint_time": "2025-04-04 15:38:46.307", "error": null }' + echo '{ "upstream_id": 7489358765827600768, "namespace": "default", "id": "97dc0d50-bbfb-4590-a258-d60b5457f188", "state": "finished", "checkpoint_tso": 457114209827422213, "checkpoint_time": "2025-04-04 15:38:46.307", "error": null }' { "upstream_id": 7489358765827600768, "namespace": "default", "id": "97dc0d50-bbfb-4590-a258-d60b5457f188", "state": "finished", "checkpoint_tso": 457114209827422213, "checkpoint_time": "2025-04-04 15:38:46.307", "error": null } ++ echo '{' '"upstream_id":' 7489358765827600768, '"namespace":' '"default",' '"id":' '"97dc0d50-bbfb-4590-a258-d60b5457f188",' '"state":' '"finished",' '"checkpoint_tso":' 457114209827422213, '"checkpoint_time":' '"2025-04-04' '15:38:46.307",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7489358765827600768, '"namespace":' '"default",' '"id":' '"97dc0d50-bbfb-4590-a258-d60b5457f188",' '"state":' '"finished",' '"checkpoint_tso":' 457114209827422213, '"checkpoint_time":' '"2025-04-04' '15:38:46.307",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:38:52 CST 2025] <<<<<< run test case changefeed_finish success! >>>>>> [2025/04/04 15:38:57.953 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark13000] [2025/04/04 15:38:57.953 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark13000] cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later table test.finish_mark not exists for 12-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:38:58 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041781b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114212615061516 MinTableBarrierTs:457114212877205518 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8338181a-3063-4dac-8b2e-1c8376bddd21 {"id":"8338181a-3063-4dac-8b2e-1c8376bddd21","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/capture/bc80f904-c082-40f4-9d9a-5f4070053b25 {"id":"bc80f904-c082-40f4-9d9a-5f4070053b25","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752335} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf9e 8338181a-3063-4dac-8b2e-1c8376bddd21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe00e bc80f904-c082-40f4-9d9a-5f4070053b25 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114212615061516,"min-table-barrier-ts":457114212877205518,"admin-job-type":0} /tidb/cdc/default/default/task/position/8338181a-3063-4dac-8b2e-1c8376bddd21/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bc80f904-c082-40f4-9d9a-5f4070053b25/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041781b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114212615061516 MinTableBarrierTs:457114212877205518 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8338181a-3063-4dac-8b2e-1c8376bddd21 {"id":"8338181a-3063-4dac-8b2e-1c8376bddd21","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/capture/bc80f904-c082-40f4-9d9a-5f4070053b25 {"id":"bc80f904-c082-40f4-9d9a-5f4070053b25","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752335} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf9e 8338181a-3063-4dac-8b2e-1c8376bddd21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe00e bc80f904-c082-40f4-9d9a-5f4070053b25 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114212615061516,"min-table-barrier-ts":457114212877205518,"admin-job-type":0} /tidb/cdc/default/default/task/position/8338181a-3063-4dac-8b2e-1c8376bddd21/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bc80f904-c082-40f4-9d9a-5f4070053b25/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041781b0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114212615061516 MinTableBarrierTs:457114212877205518 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8338181a-3063-4dac-8b2e-1c8376bddd21 {"id":"8338181a-3063-4dac-8b2e-1c8376bddd21","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752325} /tidb/cdc/default/__cdc_meta__/capture/bc80f904-c082-40f4-9d9a-5f4070053b25 {"id":"bc80f904-c082-40f4-9d9a-5f4070053b25","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752335} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fdf9e 8338181a-3063-4dac-8b2e-1c8376bddd21 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe00e bc80f904-c082-40f4-9d9a-5f4070053b25 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114212615061516,"min-table-barrier-ts":457114212877205518,"admin-job-type":0} /tidb/cdc/default/default/task/position/8338181a-3063-4dac-8b2e-1c8376bddd21/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/bc80f904-c082-40f4-9d9a-5f4070053b25/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "8338181a-3063-4dac-8b2e-1c8376bddd21" | grep id VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359234149178879 TiDB Cluster ID. tikv_gc_leader_uuid 657fef455cc000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:6335, start at 2025-04-04 15:38:56.770368983 +0800 CST m=+2.348467130 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:56.780 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:56.817 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:56.817 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359234149178879 TiDB Cluster ID. tikv_gc_leader_uuid 657fef455cc000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:6335, start at 2025-04-04 15:38:56.770368983 +0800 CST m=+2.348467130 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:56.780 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:56.817 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:56.817 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359234225741510 TiDB Cluster ID. tikv_gc_leader_uuid 657fef454ec000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:6407, start at 2025-04-04 15:38:56.742220039 +0800 CST m=+2.210073509 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:56.751 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:56.749 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:56.749 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 83-th check, retry later table batch_add_table.finish_mark not exists for 6-th check, retry later [Fri Apr 4 15:38:59 CST 2025] <<<<<< START cdc server in consistent_replicate_nfs case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_nfs.83398341.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.table2 exists check diff failed 1-th time, retry later "cluster-id": "default" "id": "bc80f904-c082-40f4-9d9a-5f4070053b25", "cluster-id": "default" run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later [2025/04/04 15:38:59.953 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] [2025/04/04 15:39:00.041 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark13000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:00.043 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test.finish_mark not exists for 13-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359245429450174 TiDB Cluster ID. tikv_gc_leader_uuid 657fef470d00006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:6713, start at 2025-04-04 15:38:58.533382765 +0800 CST m=+2.265546001 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:58.542 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:58.534 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:58.534 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359245429450174 TiDB Cluster ID. tikv_gc_leader_uuid 657fef470d00006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:6713, start at 2025-04-04 15:38:58.533382765 +0800 CST m=+2.265546001 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:58.542 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:58.534 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:58.534 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359243271922708 TiDB Cluster ID. tikv_gc_leader_uuid 657fef47194000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-zxhg8-9pc98-xkvr6, pid:6765, start at 2025-04-04 15:38:58.571105046 +0800 CST m=+2.227709949 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:40:58.579 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:38:58.583 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:28:58.583 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/common_1/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/common_1/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/common_1/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } found first match line: 457114213288771600 : [2025/04/04 15:39:00.636 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=160] [jobID=161] [query="CREATE TABLE IF NOT EXISTS `finishmark13000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114213288771600] [2025/04/04 15:39:01.037 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark13000] [2025/04/04 15:39:01.037 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114213288771600] [2025/04/04 15:39:01.040 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:01.043 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:01.047 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:01.053 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table ddl_manager.finish_mark not exists for 84-th check, retry later table batch_add_table.finish_mark not exists for 7-th check, retry later capture_id: bc80f904-c082-40f4-9d9a-5f4070053b25 check diff successfully [Fri Apr 4 15:39:00 CST 2025] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.77267728.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully [2025/04/04 15:39:01.944 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark14000] [2025/04/04 15:39:01.944 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark14000] table processor_delay.t22 exists table processor_delay.t23 not exists for 1-th check, retry later table test.finish_mark not exists for 14-th check, retry later Resume changefeed with checkpointTs3 457114213891440644 table test.table5 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:02 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7334c48b-cfd7-4026-b5bd-7d5c33949522 {"id":"7334c48b-cfd7-4026-b5bd-7d5c33949522","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752339} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcf46ad6 7334c48b-cfd7-4026-b5bd-7d5c33949522 /tidb/cdc/default/default/upstream/7489359229107577433 {"id":7489359229107577433,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7334c48b-cfd7-4026-b5bd-7d5c33949522 {"id":"7334c48b-cfd7-4026-b5bd-7d5c33949522","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752339} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcf46ad6 7334c48b-cfd7-4026-b5bd-7d5c33949522 /tidb/cdc/default/default/upstream/7489359229107577433 {"id":7489359229107577433,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7334c48b-cfd7-4026-b5bd-7d5c33949522 {"id":"7334c48b-cfd7-4026-b5bd-7d5c33949522","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752339} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcf46ad6 7334c48b-cfd7-4026-b5bd-7d5c33949522 /tidb/cdc/default/default/upstream/7489359229107577433 {"id":7489359229107577433,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ***************** properties ***************** "mysql.user"="root" "dotransactions"="false" "threadcount"="10" "scanproportion"="0" "workload"="core" "readallfields"="true" "mysql.db"="consistent_replicate_nfs" "requestdistribution"="uniform" "readproportion"="0" "recordcount"="50000" "insertproportion"="0" "mysql.host"="127.0.0.1" "fieldcount"="3" "operationcount"="0" "mysql.port"="4000" "updateproportion"="0" ********************************************** table ddl_manager.finish_mark not exists for 85-th check, retry later table batch_add_table.finish_mark not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later table test.finish_mark not exists for 15-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:03 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dff65991-f164-46f8-bf97-d5125641c298 {"id":"dff65991-f164-46f8-bf97-d5125641c298","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcfc39d6 dff65991-f164-46f8-bf97-d5125641c298 /tidb/cdc/default/default/upstream/7489359234149178879 {"id":7489359234149178879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dff65991-f164-46f8-bf97-d5125641c298 {"id":"dff65991-f164-46f8-bf97-d5125641c298","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcfc39d6 dff65991-f164-46f8-bf97-d5125641c298 /tidb/cdc/default/default/upstream/7489359234149178879 {"id":7489359234149178879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dff65991-f164-46f8-bf97-d5125641c298 {"id":"dff65991-f164-46f8-bf97-d5125641c298","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752341} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcfc39d6 dff65991-f164-46f8-bf97-d5125641c298 /tidb/cdc/default/default/upstream/7489359234149178879 {"id":7489359234149178879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.7776.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml table test.table5 not exists for 2-th check, retry later [2025/04/04 15:39:03.944 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2025/04/04 15:39:03.951 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark14000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:03.952 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] Create changefeed successfully! ID: 805dd27e-19b6-4878-bce8-4837d28b3d44 Info: {"upstream_id":7489359234149178879,"namespace":"default","id":"805dd27e-19b6-4878-bce8-4837d28b3d44","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:39:04.311570436+08:00","start_ts":457114214519275526,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_name*"],"ignore_event":["rename table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114214519275526,"checkpoint_ts":457114214519275526,"checkpoint_time":"2025-04-04 15:39:04.205"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.cli.8037.out cli tso query --pd=http://127.0.0.1:2379 found first match line: 457114214370115599 : [2025/04/04 15:39:04.566 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=162] [jobID=163] [query="CREATE TABLE IF NOT EXISTS `finishmark14000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114214370115599] [2025/04/04 15:39:04.947 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark14000] [2025/04/04 15:39:04.947 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114214370115599] [2025/04/04 15:39:04.950 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:04.954 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:05.042 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:05.047 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table ddl_manager.finish_mark exists check diff successfully table batch_add_table.finish_mark not exists for 9-th check, retry later check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... table processor_delay.t24 exists table processor_delay.t25 not exists for 1-th check, retry later table test.finish_mark not exists for 16-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... table event_filter.t1 does not exists table event_filter.t1 not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 2-th time... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 3-th time... + set +x + tso='457114214736592897 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114214736592897 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:39:06 CST 2025] <<<<<< START cdc server in common_1 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.80738075.out server --log-file /tmp/tidb_cdc_test/common_1/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/common_1/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:39:06.736 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark15000] [2025/04/04 15:39:06.736 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark15000] table test.table5 exists table test.table6 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark not exists for 10-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:39:07 CST 2025] <<<<<< run test case ddl_manager success! >>>>>> check diff failed 2-th time, retry later table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later Starting Upstream TiDB... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:07 CST 2025] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table event_filter.t1 exists Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... table event_filter.t_normal not exists for 1-th check, retry later table test.finish_mark not exists for 17-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:39:08.736 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] [2025/04/04 15:39:08.742 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark15000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:08.744 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test.table6 exists check diff failed 1-th time, retry later table batch_add_table.finish_mark not exists for 11-th check, retry later check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:09 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/370614b3-ada9-4066-916a-14a1fd03f2bd {"id":"370614b3-ada9-4066-916a-14a1fd03f2bd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752346} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcff79f3 370614b3-ada9-4066-916a-14a1fd03f2bd /tidb/cdc/default/default/upstream/7489359245429450174 {"id":7489359245429450174,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/370614b3-ada9-4066-916a-14a1fd03f2bd {"id":"370614b3-ada9-4066-916a-14a1fd03f2bd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752346} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcff79f3 370614b3-ada9-4066-916a-14a1fd03f2bd /tidb/cdc/default/default/upstream/7489359245429450174 {"id":7489359245429450174,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/370614b3-ada9-4066-916a-14a1fd03f2bd {"id":"370614b3-ada9-4066-916a-14a1fd03f2bd","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752346} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcff79f3 370614b3-ada9-4066-916a-14a1fd03f2bd /tidb/cdc/default/default/upstream/7489359245429450174 {"id":7489359245429450174,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 34628ed0-3b8c-4f10-9239-039c8b700bba Info: {"upstream_id":7489359245429450174,"namespace":"default","id":"34628ed0-3b8c-4f10-9239-039c8b700bba","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:39:09.727118877+08:00","start_ts":457114214736592897,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114214736592897,"checkpoint_ts":457114214736592897,"checkpoint_time":"2025-04-04 15:39:05.034"} table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later table test.finish_mark not exists for 18-th check, retry later table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) empty select id, val from test.availability1 where id=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table batch_add_table.finish_mark not exists for 12-th check, retry later check diff failed 4-th time, retry later [2025/04/04 15:39:10.743 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark16000] [2025/04/04 15:39:10.743 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark16000] check diff successfully Resume changefeed with checkpointTs1 457114212069277699 table test.table3 not exists for 1-th check, retry later [2025/04/04 15:39:11.546 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] [2025/04/04 15:39:11.553 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark15000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:11.555 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test.finish_mark not exists for 19-th check, retry later table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) found first match line: 457114215680835598 : [2025/04/04 15:39:09.736 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=164] [jobID=165] [query="CREATE TABLE IF NOT EXISTS `finishmark15000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114215680835598] [2025/04/04 15:39:12.345 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark15000] [2025/04/04 15:39:12.345 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114215680835598] [2025/04/04 15:39:12.347 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:12.436 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:12.441 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:12.446 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later [2025/04/04 15:39:12.743 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later INSERT - Takes(s): 10.0, Count: 20391, OPS: 2032.2, Avg(us): 4861, Min(us): 838, Max(us): 102102, 95th(us): 9000, 99th(us): 86000 table batch_add_table.finish_mark exists check diff successfully check diff successfully wait process cdc.test exit for 1-th time... table test.table3 not exists for 2-th check, retry later table test.finish_mark not exists for 20-th check, retry later table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table event_filter.t_alter exists table event_filter.t_name not exists for 1-th check, retry later nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359305776962557 TiDB Cluster ID. tikv_gc_leader_uuid 657fef54d40000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4, pid:4019, start at 2025-04-04 15:39:12.631088638 +0800 CST m=+2.259729874 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:12.639 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:12.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:12.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359305776962557 TiDB Cluster ID. tikv_gc_leader_uuid 657fef54d40000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4, pid:4019, start at 2025-04-04 15:39:12.631088638 +0800 CST m=+2.259729874 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:12.639 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:12.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:12.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359306070732691 TiDB Cluster ID. tikv_gc_leader_uuid 657fef55060000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qth0n-2cmqp-3qdk4, pid:4094, start at 2025-04-04 15:39:12.822800351 +0800 CST m=+2.326695372 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:12.834 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:12.834 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:12.834 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:39:14.749 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] [2025/04/04 15:39:14.847 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark16000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:14.848 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test.table3 not exists for 3-th check, retry later found first match line: 457114216781840399 : [2025/04/04 15:39:13.643 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=166] [jobID=167] [query="CREATE TABLE IF NOT EXISTS `finishmark16000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114216781840399] [2025/04/04 15:39:15.641 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark16000] [2025/04/04 15:39:15.641 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114216781840399] [2025/04/04 15:39:15.645 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:15.648 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:15.652 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:15.739 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table test.finish_mark not exists for 21-th check, retry later table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:39:16.150 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark17000] [2025/04/04 15:39:16.150 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark17000] table event_filter.t_name exists table event_filter.t_name1 not exists for 1-th check, retry later empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:39:16 CST 2025] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.54425444.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.table3 not exists for 4-th check, retry later table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) check diff failed 1-th time, retry later [Pipeline] // cache [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // dir table common_1.v1 exists table common_1.recover_and_insert not exists for 1-th check, retry later [Pipeline] } [2025/04/04 15:39:18.150 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] [2025/04/04 15:39:18.241 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark17000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:18.243 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test.finish_mark not exists for 22-th check, retry later table event_filter.t_name1 exists table event_filter.t_name2 not exists for 1-th check, retry later [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table common_1.recover_and_insert not exists for 2-th check, retry later [Pipeline] // podTemplate nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] } [Pipeline] // withEnv [Pipeline] } table test.table3 exists table test.table4 exists [Pipeline] // stage [Pipeline] } found first match line: 457114218033577999 : [2025/04/04 15:39:18.736 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=168] [jobID=169] [query="CREATE TABLE IF NOT EXISTS `finishmark17000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114218033577999] [2025/04/04 15:39:19.344 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark17000] [2025/04/04 15:39:19.344 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114218033577999] [2025/04/04 15:39:19.347 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:19.351 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:19.355 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:19.437 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] check diff failed 1-th time, retry later check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:20 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d {"id":"2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752357} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd3a77da 2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d /tidb/cdc/default/default/upstream/7489359305776962557 {"id":7489359305776962557,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d {"id":"2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752357} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd3a77da 2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d /tidb/cdc/default/default/upstream/7489359305776962557 {"id":7489359305776962557,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d {"id":"2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752357} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd3a77da 2152cf0d-b574-4fe3-82b1-2f2ac35a0e3d /tidb/cdc/default/default/upstream/7489359305776962557 {"id":7489359305776962557,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table test.finish_mark not exists for 23-th check, retry later table event_filter.t_name2 exists table event_filter.t_name3 not exists for 1-th check, retry later Create changefeed successfully! ID: 58e03469-45b6-4af1-bb1b-59e851cd8b05 Info: {"upstream_id":7489359305776962557,"namespace":"default","id":"58e03469-45b6-4af1-bb1b-59e851cd8b05","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:39:20.152660359+08:00","start_ts":457114217844047874,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114217844047874,"checkpoint_ts":457114217844047874,"checkpoint_time":"2025-04-04 15:39:16.888"} table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later [2025/04/04 15:39:20.342 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark18000] [2025/04/04 15:39:20.342 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark18000] The 1 times to try to start tidb cluster... table force_replicate_table.t0 not exists for 1-th check, retry later nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table common_1.recover_and_insert not exists for 3-th check, retry later check diff failed 3-th time, retry later table test.finish_mark not exists for 24-th check, retry later table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... [2025/04/04 15:39:22.343 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] [2025/04/04 15:39:22.350 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark18000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:22.351 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table force_replicate_table.t0 exists table force_replicate_table.t1 not exists for 1-th check, retry later table event_filter.t_name3 exists wait process cdc.test exit for 2-th time... INSERT - Takes(s): 20.0, Count: 30017, OPS: 1501.2, Avg(us): 6629, Min(us): 838, Max(us): 102102, 95th(us): 71000, 99th(us): 89000 empty select id, val from test.availability3 where id=1 run task successfully test_gap_between_watch_capture pass table event_filter.finish_mark exists check diff failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:23 CST 2025] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.86648666.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table common_1.recover_and_insert exists table common_1.finish_mark not exists for 1-th check, retry later check diff failed 4-th time, retry later wait process cdc.test exit for 1-th time... found first match line: 457114219141136398 : [2025/04/04 15:39:22.736 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=170] [jobID=171] [query="CREATE TABLE IF NOT EXISTS `finishmark18000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114219141136398] [2025/04/04 15:39:23.465 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark18000] [2025/04/04 15:39:23.465 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114219141136398] [2025/04/04 15:39:23.538 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:23.541 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:23.545 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:23.549 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] start tidb cluster in /tmp/tidb_cdc_test/multi_source Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.finish_mark not exists for 25-th check, retry later table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:39:23.713 +08:00] [WARN] [diff.go:180] ["table struct is not equal"] [reason="column num not equal, one is 2 another is 6"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case delete_owner_key [Fri Apr 4 15:39:24 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.47544756.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory table force_replicate_table.t1 exists table force_replicate_table.t2 not exists for 1-th check, retry later table common_1.finish_mark not exists for 2-th check, retry later [2025/04/04 15:39:25.346 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark19000] [2025/04/04 15:39:25.346 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark19000] check diff failed 2-th time, retry later table processor_delay.t34 exists table processor_delay.t35 not exists for 1-th check, retry later check diff failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.finish_mark not exists for 26-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:26 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3376 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b746a710-b2e9-4355-afa6-f2bef6304013 {"id":"b746a710-b2e9-4355-afa6-f2bef6304013","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752363} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcd78aa5 b746a710-b2e9-4355-afa6-f2bef6304013 /tidb/cdc/default/default/changefeed/info/d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d {"upstream-id":7489359191455856322,"namespace":"default","changefeed-id":"d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:38:54.774015741+08:00","start-ts":457114212069277699,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"failed","error":{"time":"2025-04-04T15:39:23.692909682+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114219619024902} /tidb/cdc/default/default/changefeed/status/d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d {"checkpoint-ts":457114213904285718,"min-table-barrier-ts":457114213930500104,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489359191455856322 {"id":7489359191455856322,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b746a710-b2e9-4355-afa6-f2bef6304013 {"id":"b746a710-b2e9-4355-afa6-f2bef6304013","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752363} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcd78aa5 b746a710-b2e9-4355-afa6-f2bef6304013 /tidb/cdc/default/default/changefeed/info/d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d {"upstream-id":7489359191455856322,"namespace":"default","changefeed-id":"d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:38:54.774015741+08:00","start-ts":457114212069277699,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"failed","error":{"time":"2025-04-04T15:39:23.692909682+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114219619024902} /tidb/cdc/default/default/changefeed/status/d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d {"checkpoint-ts":457114213904285718,"min-table-barrier-ts":457114213930500104,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489359191455856322 {"id":7489359191455856322,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b746a710-b2e9-4355-afa6-f2bef6304013 {"id":"b746a710-b2e9-4355-afa6-f2bef6304013","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752363} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbcd78aa5 b746a710-b2e9-4355-afa6-f2bef6304013 /tidb/cdc/default/default/changefeed/info/d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d {"upstream-id":7489359191455856322,"namespace":"default","changefeed-id":"d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:38:54.774015741+08:00","start-ts":457114212069277699,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"failed","error":{"time":"2025-04-04T15:39:23.692909682+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114219619024902} /tidb/cdc/default/default/changefeed/status/d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d {"checkpoint-ts":457114213904285718,"min-table-barrier-ts":457114213930500104,"admin-job-type":1} /tidb/cdc/default/default/upstream/7489359191455856322 {"id":7489359191455856322,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d -s + info='{ "upstream_id": 7489359191455856322, "namespace": "default", "id": "d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d", "state": "failed", "checkpoint_tso": 457114213904285718, "checkpoint_time": "2025-04-04 15:39:01.859", "error": { "time": "2025-04-04T15:39:23.692909682+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7489359191455856322, "namespace": "default", "id": "d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d", "state": "failed", "checkpoint_tso": 457114213904285718, "checkpoint_time": "2025-04-04 15:39:01.859", "error": { "time": "2025-04-04T15:39:23.692909682+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7489359191455856322, "namespace": "default", "id": "d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d", "state": "failed", "checkpoint_tso": 457114213904285718, "checkpoint_time": "2025-04-04 15:39:01.859", "error": { "time": "2025-04-04T15:39:23.692909682+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ echo '{' '"upstream_id":' 7489359191455856322, '"namespace":' '"default",' '"id":' '"d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d",' '"state":' '"failed",' '"checkpoint_tso":' 457114213904285718, '"checkpoint_time":' '"2025-04-04' '15:39:01.859",' '"error":' '{' '"time":' '"2025-04-04T15:39:23.692909682+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .state + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7489359191455856322, '"namespace":' '"default",' '"id":' '"d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d",' '"state":' '"failed",' '"checkpoint_tso":' 457114213904285718, '"checkpoint_time":' '"2025-04-04' '15:39:01.859",' '"error":' '{' '"time":' '"2025-04-04T15:39:23.692909682+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully table force_replicate_table.t2 exists table force_replicate_table.t3 not exists for 1-th check, retry later check_changefeed_state http://127.0.0.1:2379 d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d -s + info='{ "upstream_id": 7489359191455856322, "namespace": "default", "id": "d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d", "state": "normal", "checkpoint_tso": 457114220418564100, "checkpoint_time": "2025-04-04 15:39:26.709", "error": null }' + echo '{ "upstream_id": 7489359191455856322, "namespace": "default", "id": "d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d", "state": "normal", "checkpoint_tso": 457114220418564100, "checkpoint_time": "2025-04-04 15:39:26.709", "error": null }' { "upstream_id": 7489359191455856322, "namespace": "default", "id": "d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d", "state": "normal", "checkpoint_tso": 457114220418564100, "checkpoint_time": "2025-04-04 15:39:26.709", "error": null } ++ echo '{' '"upstream_id":' 7489359191455856322, '"namespace":' '"default",' '"id":' '"d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d",' '"state":' '"normal",' '"checkpoint_tso":' 457114220418564100, '"checkpoint_time":' '"2025-04-04' '15:39:26.709",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359191455856322, '"namespace":' '"default",' '"id":' '"d56ca50d-36ea-4d9d-b04f-ef6c8fbdae7d",' '"state":' '"normal",' '"checkpoint_tso":' 457114220418564100, '"checkpoint_time":' '"2025-04-04' '15:39:26.709",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table common_1.finish_mark not exists for 3-th check, retry later table test1.table1 not exists for 1-th check, retry later [2025/04/04 15:39:27.346 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark19000] [2025/04/04 15:39:27.441 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark19000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:27.442 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] check diff successfully table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:27 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004415440 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114219181768706 MinTableBarrierTs:457114220492488709 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b {"id":"4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752364} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe140 4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114219181768706,"min-table-barrier-ts":457114220492488709,"admin-job-type":0} /tidb/cdc/default/default/task/position/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004415440 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114219181768706 MinTableBarrierTs:457114220492488709 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b {"id":"4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752364} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe140 4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114219181768706,"min-table-barrier-ts":457114220492488709,"admin-job-type":0} /tidb/cdc/default/default/task/position/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004415440 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114219181768706 MinTableBarrierTs:457114220492488709 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114219181768706, checkpointTs: 457114219181768706, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b {"id":"4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752364} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe140 4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114219181768706,"min-table-barrier-ts":457114220492488709,"admin-job-type":0} /tidb/cdc/default/default/task/position/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 27-th check, retry later found first match line: 457114220425641998 : [2025/04/04 15:39:27.836 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=172] [jobID=173] [query="CREATE TABLE IF NOT EXISTS `finishmark19000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114220425641998] [2025/04/04 15:39:28.561 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark19000] [2025/04/04 15:39:28.561 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114220425641998] [2025/04/04 15:39:28.637 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:28.649 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:28.653 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:28.658 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] wait process cdc.test exit for 2-th time... table force_replicate_table.t3 exists table force_replicate_table.t4 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:29 CST 2025] <<<<<< run test case event_filter success! >>>>>> table common_1.finish_mark not exists for 4-th check, retry later table test1.table1 not exists for 2-th check, retry later "is-owner": true, run task successfully [2025/04/04 15:39:29.640 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20001] [2025/04/04 15:39:29.640 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:39:29.641 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20002] [2025/04/04 15:39:29.641 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:39:29.644 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20003] [2025/04/04 15:39:29.644 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:39:29.644 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20004] [2025/04/04 15:39:29.644 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:39:29.664 +08:00] [INFO] [case.go:544] ["upstream create table executed"] [tblName=finishmark20000] [2025/04/04 15:39:29.664 +08:00] [INFO] [case.go:561] ["one upstream workload finished"] [round=20004] [2025/04/04 15:39:29.664 +08:00] [INFO] [case.go:576] ["downstream start wait for table"] [tblName=finishmark20000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 28-th check, retry later table processor_delay.t36 exists table processor_delay.t37 not exists for 1-th check, retry later owner pid: 4759 owner id 4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b default owner key /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe140 [Fri Apr 4 15:39:30 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.48744876.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later table common_1.finish_mark exists check diff successfully [2025/04/04 15:39:31.665 +08:00] [INFO] [case.go:687] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark20000] [2025/04/04 15:39:31.673 +08:00] [INFO] [case.go:697] ["try parse finishedTs from ticdc log"] [tableName=finishmark20000] /tmp/tidb_cdc_test/bank/cdcdown.log [2025/04/04 15:39:31.674 +08:00] [INFO] [case.go:717] ["total files"] [file="[/tmp/tidb_cdc_test/bank/cdcdown.log]"] table test1.table1 exists table test1.table2 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... found first match line: 457114221520355343 : [2025/04/04 15:39:31.666 +08:00] [INFO] [ddl_manager.go:298] ["handle a ddl job"] [namespace=default] [changefeed=changefeed-for-find-finished-ts] [tableID=174] [jobID=175] [query="CREATE TABLE IF NOT EXISTS `finishmark20000` (`foo` BIGINT PRIMARY KEY)"] [finishedTs=457114221520355343] [2025/04/04 15:39:31.866 +08:00] [INFO] [case.go:581] ["ddl synced"] [table=finishmark20000] [2025/04/04 15:39:31.866 +08:00] [INFO] [case.go:587] ["downstream sync success"] [endTs=457114221520355343] [2025/04/04 15:39:31.869 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:31.873 +08:00] [INFO] [case.go:191] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:31.876 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2025/04/04 15:39:31.880 +08:00] [INFO] [case.go:308] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2025/04/04 15:39:31.881 +08:00] [INFO] [case.go:608] ["one downstream exit due to receive done"] [2025/04/04 15:39:31.881 +08:00] [INFO] [case.go:620] ["bank test finished"] [valid=20] [tries=20] [ratio=1] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table test.finish_mark not exists for 29-th check, retry later table testSync.usertable exists table testSync.simple1 exists table testSync.simple2 exists + cleanup_process cdc.test table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359379822393222 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6694c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:6897, start at 2025-04-04 15:39:30.807436023 +0800 CST m=+2.800733524 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:30.815 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:30.857 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:30.857 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359379822393222 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6694c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:6897, start at 2025-04-04 15:39:30.807436023 +0800 CST m=+2.800733524 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:30.815 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:30.857 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:30.857 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359384128920818 TiDB Cluster ID. tikv_gc_leader_uuid 657fef66c580005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:6970, start at 2025-04-04 15:39:30.979935217 +0800 CST m=+2.891634392 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:30.989 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:30.966 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:30.966 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_source/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_source/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... INSERT - Takes(s): 30.0, Count: 41353, OPS: 1376.9, Avg(us): 7225, Min(us): 838, Max(us): 102102, 95th(us): 79000, 99th(us): 89000 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:32 CST 2025] <<<<<< run test case common_1 success! >>>>>> table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit ++ date + echo '[Fri Apr 4 15:39:33 CST 2025] <<<<<< run test case bank success! >>>>>>' [Fri Apr 4 15:39:33 CST 2025] <<<<<< run test case bank success! >>>>>> + stop_tidb_cluster table test1.table2 exists check diff successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:33 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003953290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114222065352712 MinTableBarrierTs:457114222065352712 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4b759252-a648-4f2f-b44d-da224585a2d4 {"id":"4b759252-a648-4f2f-b44d-da224585a2d4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752371} /tidb/cdc/default/__cdc_meta__/capture/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b {"id":"4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752364} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe140 4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe18c 4b759252-a648-4f2f-b44d-da224585a2d4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114222065352712,"min-table-barrier-ts":457114222065352712,"admin-job-type":0} /tidb/cdc/default/default/task/position/4b759252-a648-4f2f-b44d-da224585a2d4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003953290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114222065352712 MinTableBarrierTs:457114222065352712 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4b759252-a648-4f2f-b44d-da224585a2d4 {"id":"4b759252-a648-4f2f-b44d-da224585a2d4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752371} /tidb/cdc/default/__cdc_meta__/capture/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b {"id":"4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752364} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe140 4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe18c 4b759252-a648-4f2f-b44d-da224585a2d4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114222065352712,"min-table-barrier-ts":457114222065352712,"admin-job-type":0} /tidb/cdc/default/default/task/position/4b759252-a648-4f2f-b44d-da224585a2d4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003953290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114222065352712 MinTableBarrierTs:457114222065352712 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4b759252-a648-4f2f-b44d-da224585a2d4 {"id":"4b759252-a648-4f2f-b44d-da224585a2d4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752371} /tidb/cdc/default/__cdc_meta__/capture/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b {"id":"4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752364} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe140 4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe18c 4b759252-a648-4f2f-b44d-da224585a2d4 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114222065352712,"min-table-barrier-ts":457114222065352712,"admin-job-type":0} /tidb/cdc/default/default/task/position/4b759252-a648-4f2f-b44d-da224585a2d4/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} + grep -q 'etcd info' /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "4ecb804e-3a8f-412a-a6a4-d8c8e1ac6f4b default" | grep id check diff failed 2-th time, retry later table test.finish_mark not exists for 30-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t38 exists table processor_delay.t39 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359293518727697 TiDB Cluster ID. tikv_gc_leader_uuid 657fef681b80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:10135, start at 2025-04-04 15:39:32.345019223 +0800 CST m=+24.524832848 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:32.352 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:32.334 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:32.334 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359293518727697 TiDB Cluster ID. tikv_gc_leader_uuid 657fef681b80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:10135, start at 2025-04-04 15:39:32.345019223 +0800 CST m=+24.524832848 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:32.352 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:32.334 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:32.334 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359297427746394 TiDB Cluster ID. tikv_gc_leader_uuid 657fef51fd0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-djxkt-cmhg4-27044, pid:10199, start at 2025-04-04 15:39:09.70775373 +0800 CST m=+1.815285808 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:09.715 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:09.684 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:09.684 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.8199.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 1-th time... table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 "id": "4b759252-a648-4f2f-b44d-da224585a2d4", run task successfully wait process cdc.test exit for 2-th time... run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:36 CST 2025] <<<<<< run test case changefeed_resume_with_checkpoint_ts success! >>>>>> run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11480.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 3-th time, retry later table test.finish_mark not exists for 31-th check, retry later + set +x + tso='457114222545076225 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114222545076225 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:39:36 CST 2025] <<<<<< START cdc server in multi_source case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.82408242.out server --log-file /tmp/tidb_cdc_test/multi_source/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_source/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 capture_id: 4b759252-a648-4f2f-b44d-da224585a2d4 1 ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '4b759252-a648-4f2f-b44d-da224585a2d4' 4b759252-a648-4f2f-b44d-da224585a2d4 run task successfully curl -X GET http://127.0.0.1:8300/status | grep '"is_owner": false' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 30500 0 --:--:-- --:--:-- --:--:-- 30500 "is_owner": false run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 + set +x + tso='457114223033188353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114223033188353 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:39:38 CST 2025] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1150511507.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later table test.finish_mark not exists for 32-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359403736099085 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6bf24000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:6587, start at 2025-04-04 15:39:36.281406692 +0800 CST m=+2.349123729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:36.326 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:36.325 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:36.325 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359403736099085 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6bf24000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:6587, start at 2025-04-04 15:39:36.281406692 +0800 CST m=+2.349123729 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:36.326 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:36.325 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:36.325 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359402995735401 TiDB Cluster ID. tikv_gc_leader_uuid 657fef6c23c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:6651, start at 2025-04-04 15:39:36.479438621 +0800 CST m=+2.440456234 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:36.488 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:36.463 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:36.463 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t40 exists table processor_delay.t41 not exists for 1-th check, retry later run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sql_mode/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/sql_mode + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + rm -rf /tmp/tidb_cdc_test/sql_mode + mkdir -p /tmp/tidb_cdc_test/sql_mode + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:39 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/add84734-4265-47f2-8060-68177241c201 {"id":"add84734-4265-47f2-8060-68177241c201","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752376} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd7f57e3 add84734-4265-47f2-8060-68177241c201 /tidb/cdc/default/default/upstream/7489359379822393222 {"id":7489359379822393222,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/add84734-4265-47f2-8060-68177241c201 {"id":"add84734-4265-47f2-8060-68177241c201","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752376} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd7f57e3 add84734-4265-47f2-8060-68177241c201 /tidb/cdc/default/default/upstream/7489359379822393222 {"id":7489359379822393222,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/add84734-4265-47f2-8060-68177241c201 {"id":"add84734-4265-47f2-8060-68177241c201","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752376} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd7f57e3 add84734-4265-47f2-8060-68177241c201 /tidb/cdc/default/default/upstream/7489359379822393222 {"id":7489359379822393222,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.8293.out cli changefeed create --start-ts=457114222545076225 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Create changefeed successfully! ID: 0721ab4f-f884-4ec9-a084-81ddda395963 Info: {"upstream_id":7489359379822393222,"namespace":"default","id":"0721ab4f-f884-4ec9-a084-81ddda395963","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:39:39.900862101+08:00","start_ts":457114222545076225,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114222545076225,"checkpoint_ts":457114222545076225,"checkpoint_time":"2025-04-04 15:39:34.821"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 5-th time, retry later table test.finish_mark not exists for 33-th check, retry later [Fri Apr 4 15:39:40 CST 2025] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.78867888.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later Run finished, takes 37.44964653s INSERT - Takes(s): 37.4, Count: 50000, OPS: 1335.3, Avg(us): 7459, Min(us): 838, Max(us): 102102, 95th(us): 79000, 99th(us): 89000 table consistent_replicate_nfs.usertable exists wait process cdc.test exit for 1-th time... pass check, checkpoint tso not forward after 10s run task successfully wait process 8256 exit for 1-th time... + set +x go: downloading go1.23.4 (linux/amd64) wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:41 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/424b847f-1739-4123-b4d6-c763d4394bb4 {"id":"424b847f-1739-4123-b4d6-c763d4394bb4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752378} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd2e8a4b 424b847f-1739-4123-b4d6-c763d4394bb4 /tidb/cdc/default/default/upstream/7489359293518727697 {"id":7489359293518727697,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/424b847f-1739-4123-b4d6-c763d4394bb4 {"id":"424b847f-1739-4123-b4d6-c763d4394bb4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752378} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd2e8a4b 424b847f-1739-4123-b4d6-c763d4394bb4 /tidb/cdc/default/default/upstream/7489359293518727697 {"id":7489359293518727697,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/424b847f-1739-4123-b4d6-c763d4394bb4 {"id":"424b847f-1739-4123-b4d6-c763d4394bb4","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752378} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd2e8a4b 424b847f-1739-4123-b4d6-c763d4394bb4 /tidb/cdc/default/default/upstream/7489359293518727697 {"id":7489359293518727697,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11559.out cli changefeed create --start-ts=457114223033188353 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 2-th time, retry later Create changefeed successfully! ID: b133e2f5-478c-48d4-87d6-0381efa2b7f4 Info: {"upstream_id":7489359293518727697,"namespace":"default","id":"b133e2f5-478c-48d4-87d6-0381efa2b7f4","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:39:41.658417074+08:00","start_ts":457114223033188353,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114223033188353,"checkpoint_ts":457114223033188353,"checkpoint_time":"2025-04-04 15:39:36.683"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process 8256 exit for 2-th time... wait process cdc.test exit for 3-th time... table test.finish_mark exists start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table processor_delay.t42 exists table processor_delay.t43 not exists for 1-th check, retry later wait process 8256 exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit check diff successfully check diff successfully wait process 8256 exit for 4-th time... + set +x wait process cdc.test exit for 1-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kill_owner_with_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8256) - No such process wait process 8256 exit for 5-th time... process 8256 already exit [Fri Apr 4 15:39:43 CST 2025] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.88728874.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:43 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c642d095-66b1-4749-99cd-46790a9ae9f5 {"id":"c642d095-66b1-4749-99cd-46790a9ae9f5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752380} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8ec1d8 c642d095-66b1-4749-99cd-46790a9ae9f5 /tidb/cdc/default/default/upstream/7489359403736099085 {"id":7489359403736099085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c642d095-66b1-4749-99cd-46790a9ae9f5 {"id":"c642d095-66b1-4749-99cd-46790a9ae9f5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752380} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8ec1d8 c642d095-66b1-4749-99cd-46790a9ae9f5 /tidb/cdc/default/default/upstream/7489359403736099085 {"id":7489359403736099085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c642d095-66b1-4749-99cd-46790a9ae9f5 {"id":"c642d095-66b1-4749-99cd-46790a9ae9f5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752380} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbd8ec1d8 c642d095-66b1-4749-99cd-46790a9ae9f5 /tidb/cdc/default/default/upstream/7489359403736099085 {"id":7489359403736099085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.7938.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/batch_add_table/conf/changefeed.toml cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:43 CST 2025] <<<<<< run test case tidb_mysql_test success! >>>>>> table processor_delay.t43 exists Create changefeed successfully! table processor_delay.t44 not exists for 1-th check, retry later ID: 1861ec46-754f-4745-96b9-1485d39f263a Info: {"upstream_id":7489359403736099085,"namespace":"default","id":"1861ec46-754f-4745-96b9-1485d39f263a","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:39:43.943938003+08:00","start_ts":457114224902799364,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*","!test.t_1"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114224902799364,"checkpoint_ts":457114224902799364,"checkpoint_time":"2025-04-04 15:39:43.815"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later table test.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 4-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } + set +x [Pipeline] // timeout [Pipeline] } table resourcecontrol.finish_mark not exists for 1-th check, retry later [Pipeline] // stage [Pipeline] } table test.finish_mark exists table test.t_1 not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] // container check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } check diff successfully table processor_delay.t44 exists table processor_delay.t45 not exists for 1-th check, retry later run task successfully [Pipeline] // withEnv [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // node [Pipeline] // cache [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] } go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20241223052309-3735ed55a394 go: downloading github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 start tidb cluster in /tmp/tidb_cdc_test/kill_owner_with_ddl Starting Upstream PD... [Pipeline] // podTemplate wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:46 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489359110825992106 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:37.223216981 +0800 CST StartTs:457114207417270275 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00317f4d0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114207443484674} {CheckpointTs:457114211218620420 MinTableBarrierTs:457114225321967620 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419 {"id":"314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752383} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c2983 314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489359110825992106,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:38:37.223216981+08:00","start-ts":457114207417270275,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114207443484674} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114211218620420,"min-table-barrier-ts":457114225321967620,"admin-job-type":0} /tidb/cdc/default/default/task/position/314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489359110825992106 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:37.223216981 +0800 CST StartTs:457114207417270275 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00317f4d0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114207443484674} {CheckpointTs:457114211218620420 MinTableBarrierTs:457114225321967620 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419 {"id":"314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752383} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c2983 314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489359110825992106,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:38:37.223216981+08:00","start-ts":457114207417270275,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114207443484674} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114211218620420,"min-table-barrier-ts":457114225321967620,"admin-job-type":0} /tidb/cdc/default/default/task/position/314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7489359110825992106 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2025-04-04 15:38:37.223216981 +0800 CST StartTs:457114207417270275 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00317f4d0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114207443484674} {CheckpointTs:457114211218620420 MinTableBarrierTs:457114225321967620 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114211218620420, checkpointTs: 457114211218620420, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419 {"id":"314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752383} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbc8c2983 314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7489359110825992106,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2025-04-04T15:38:37.223216981+08:00","start-ts":457114207417270275,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114207443484674} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":457114211218620420,"min-table-barrier-ts":457114225321967620,"admin-job-type":0} /tidb/cdc/default/default/task/position/314abe2a-d6c4-4cf9-ae1f-1d0c68f1d419/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359110825992106 {"id":7489359110825992106,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully check_ts_forward ddl-only-block-related-table [Pipeline] // dir check diff failed 1-th time, retry later [Pipeline] } [Pipeline] } Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] // withEnv [Pipeline] // withCredentials wait process cdc.test exit for 2-th time... [Pipeline] } [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // stage [Pipeline] // timeout [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:47 CST 2025] <<<<<< run test case many_pk_or_uk success! >>>>>> nonempty select id, val from test.availability1 where id=1 and val=22 [Pipeline] // withEnv [Pipeline] } Starting Upstream TiDB... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:47 CST 2025] <<<<<< run test case force_replicate_table success! >>>>>> [Pipeline] // node run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] } Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 2-th time, retry later table resourcecontrol.finish_mark exists check diff successfully [Pipeline] // podTemplate table test.t_1 exists table test.t_2 exists table test.t_3 exists table test.t_4 exists table test.t_5 exists table test.t_6 exists table test.t_7 exists table test.t_8 exists table test.t_9 exists table test.t_10 exists table test.t_11 exists table test.t_12 exists table test.t_13 exists table test.t_14 exists table test.t_15 exists table test.t_16 exists table test.t_17 exists table test.t_18 exists table test.t_19 exists table test.t_20 exists table test.t_21 exists table test.t_22 exists table test.t_23 exists table test.t_24 exists table test.t_25 exists table test.t_26 exists table test.t_27 exists table test.t_28 exists table test.t_29 exists table test.t_30 exists table test.t_31 exists table test.t_32 exists table test.t_33 exists table test.t_34 exists table test.t_35 exists table test.t_36 exists table test.t_37 exists table test.t_38 exists table test.t_39 exists table test.t_40 exists table test.t_41 exists table test.t_42 exists table test.t_43 exists table test.t_44 exists table test.t_45 exists table test.t_46 exists table test.t_47 exists table test.t_48 exists table test.t_49 exists table test.t_50 exists table test.t_51 exists table test.t_52 exists table test.t_53 exists table test.t_54 exists table test.t_55 exists table test.t_56 exists table test.t_57 exists table test.t_58 exists table test.t_59 exists table test.t_60 exists table test.t_61 exists table test.t_62 exists table test.t_63 exists table test.t_64 exists table test.t_65 exists table test.t_66 exists table test.t_67 exists table test.t_68 exists table test.t_69 exists table test.t_70 exists table test.t_71 exists table test.t_72 exists table test.t_73 exists table test.t_74 exists table test.t_75 exists table test.t_76 exists table test.t_77 exists table test.t_78 exists table test.t_79 exists table test.t_80 exists table test.t_81 exists table test.t_82 exists table test.t_83 exists table test.t_84 exists [Pipeline] } [Pipeline] // withEnv wait process cdc.test exit for 1-th time... [Pipeline] } table test.t_85 exists table test.t_86 exists table test.t_87 exists table test.t_88 exists table test.t_89 exists table test.t_90 exists table test.t_91 exists table test.t_92 exists table test.t_93 exists table test.t_94 exists table test.t_95 exists table test.t_96 exists table test.t_97 exists table test.t_98 exists table test.t_99 exists table test.t_100 exists [Pipeline] // stage table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later changefeed is working normally rts: 457114211218620420->457114225846255620 checkpoint: 457114211218620420->457114225846255620 run task successfully [Pipeline] } wait process cdc.test exit for 2-th time... go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20241223052309-3735ed55a394 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/tikv/pd/client v0.0.0-20241111073742-238d4d79ea31 go: downloading github.com/bits-and-blooms/bitset v1.14.3 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/pingcap/kvproto v0.0.0-20241120022153-92b0414aeed8 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/pingcap/tipb v0.0.0-20241105053214-f91fdb81a69e go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/stretchr/testify v1.10.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading cloud.google.com/go/storage v1.39.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20241209094930-06d7f4b9233b go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/grpc v1.64.0 go: downloading golang.org/x/text v0.21.0 go: downloading golang.org/x/sync v0.10.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/gorilla/mux v1.8.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/prometheus/client_golang v1.20.5 go: downloading golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.16.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/shirou/gopsutil/v3 v3.24.5 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/klauspost/compress v1.17.9 go: downloading github.com/tikv/pd v1.1.0-beta.0.20240407022249-7179657d129b go: downloading golang.org/x/time v0.7.0 go: downloading golang.org/x/sys v0.28.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading cloud.google.com/go v0.112.2 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible wait process cdc.test exit for 1-th time... go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/aws/aws-sdk-go v1.55.5 go: downloading github.com/qri-io/jsonschema v0.2.1 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/spf13/pflag v1.0.5 go: downloading golang.org/x/oauth2 v0.24.0 go: downloading google.golang.org/api v0.170.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/danjacques/gofslock v0.0.0-20240212154529-d899e02bfe22 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.3-0.20240520233950-75e935fc3e17 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/golang/snappy v0.0.4 table test.t_100 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:49 CST 2025] <<<<<< run test case resourcecontrol success! >>>>>> go: downloading github.com/google/pprof v0.0.0-20241001023024-f4c0cfd0cf1d go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading google.golang.org/protobuf v1.35.2 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.61.0 go: downloading github.com/prometheus/procfs v0.15.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.10.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 go: downloading golang.org/x/crypto v0.31.0 wait process cdc.test exit for 2-th time... empty select id, val from test.availability1 where id=1 run task successfully go: downloading golang.org/x/tools v0.28.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:39:50 CST 2025] <<<<<< run test case ddl_only_block_related_table success! >>>>>> nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/dgryski/go-farm v0.0.0-20240924180020-3414d57e47da go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading golang.org/x/term v0.27.0 go: downloading cloud.google.com/go/compute/metadata v0.3.0 go: downloading cloud.google.com/go/iam v1.1.7 go: downloading github.com/googleapis/gax-go/v2 v2.12.3 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/apache/thrift v0.16.0 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240401170217-c3f982113cda go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/qri-io/jsonpointer v0.1.1 go: downloading github.com/mattn/go-runewidth v0.0.16 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/pingcap/badger v1.5.1-0.20241015064302-38533b6cbf8d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/robfig/cron v1.2.0 table processor_delay.t46 exists table processor_delay.t47 not exists for 1-th check, retry later check diff failed 3-th time, retry later go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.24.0 go: downloading go.opentelemetry.io/otel/trace v1.24.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/fatih/color v1.18.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/apache/arrow/go/v12 v12.0.1 go: downloading github.com/goccy/go-reflect v1.2.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.1 go: downloading github.com/rivo/uniseg v0.4.7 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/redact v1.1.5 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.24.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/rogpeppe/go-internal v1.13.1 go: downloading github.com/kr/text v0.2.0 table test.t_100 not exists for 2-th check, retry later go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/zeebo/xxh3 v1.0.2 go: downloading github.com/google/flatbuffers v2.0.8+incompatible go: downloading github.com/andybalholm/brotli v1.0.5 go: downloading github.com/klauspost/cpuid/v2 v2.2.4 nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_nfs/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["consistent_replicate_nfs.usertable*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "457114223314731155" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" [2025/04/04 15:39:52.485 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:39:52.485 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:39:52.486 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:39:52.486 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:39:52.486 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114222790443052] [resolvedTs=457114223314731155] [2025/04/04 15:39:52.486 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=320.146ยตs] [2025/04/04 15:39:52.487 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:39:52.487 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:39:52.487 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:39:52.491 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:39:52.495 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:39:52.497 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:39:52.497 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:39:52.497 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:39:52.497 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:39:52.502 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:39:52.503 +08:00] [INFO] [file.go:284] ["ignore logs which commitTs is greater than resolvedTs"] [filename=127.0.0.1:8300_34fc6698-321b-41bb-b768-65b6ab1d3123_row_457114223367159919_4e91bf2a-6fa6-4610-9cd9-6aa0df7642a8.log] [endTs=457114223314731155] [2025/04/04 15:39:52.504 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=18.695566ms] [2025/04/04 15:39:52.505 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:39:52.505 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359467753540872 TiDB Cluster ID. tikv_gc_leader_uuid 657fef79994000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9146, start at 2025-04-04 15:39:50.273178944 +0800 CST m=+2.246077329 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.281 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sql_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sql_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sql_mode/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 4-th time, retry later Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:39:52.728 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}"] [checkpointTs=457114223314731155] [2025/04/04 15:39:52.728 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}"] [checkpointTs=457114223314731155] [2025/04/04 15:39:52.828 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=2351] [appliedDDLCount=0] [currentCheckpoint=457114223314731155] [2025/04/04 15:39:52.828 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:39:52.828 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:39:52.828 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:39:52.828 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:39:52.828 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:39:52.829 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] Apply redo log successfully go: downloading github.com/pierrec/lz4 v2.6.1+incompatible check diff successfully [Fri Apr 4 15:39:53 CST 2025] <<<<<< run test case consistent_replicate_nfs success! >>>>>> table test.t_100 not exists for 3-th check, retry later count(*) 46972 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_reentrant/run.sh using Sink-Type: mysql... <<================= ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tidb cluster... nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table processor_delay.t48 exists table processor_delay.t49 not exists for 1-th check, retry later + trap stop_tidb_cluster EXIT + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''NO_BACKSLASH_ESCAPES'\'';' 127.0.0.1 3306 + cd /tmp/tidb_cdc_test/sql_mode ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10392.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 table test.t_100 not exists for 4-th check, retry later table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later + set +x + tso='457114227756498945 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114227756498945 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114227756498945 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:39:56 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1041610418.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359484104304026 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7ded80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:7155, start at 2025-04-04 15:39:54.687902964 +0800 CST m=+2.221296461 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:54.696 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:54.678 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:54.678 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359484104304026 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7ded80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:7155, start at 2025-04-04 15:39:54.687902964 +0800 CST m=+2.221296461 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:54.696 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:54.678 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:54.678 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359480055202266 TiDB Cluster ID. tikv_gc_leader_uuid 657fef7df8c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:7229, start at 2025-04-04 15:39:54.7380004 +0800 CST m=+2.187836160 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:54.745 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:54.723 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:54.723 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.t_100 not exists for 5-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } start tidb cluster in /tmp/tidb_cdc_test/ddl_reentrant Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] // container [Pipeline] } table processor_delay.t50 exists [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } check diff failed 1-th time, retry later [Fri Apr 4 15:39:58 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.86148616.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // podTemplate [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7129 0 --:--:-- --:--:-- --:--:-- 7129 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:39:45.904","puller_resolved_ts":"2025-04-04 15:39:39.904","last_synced_ts":"2025-04-04 15:37:30.504","now_ts":"2025-04-04 15:39:47.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:39:45.904","puller_resolved_ts":"2025-04-04' '15:39:39.904","last_synced_ts":"2025-04-04' '15:37:30.504","now_ts":"2025-04-04' '15:39:47.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 9175 9.1 0.0 16298508 126720 ? Sl 15:37 0:14 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 9240 6.7 0.0 16222728 124416 ? Sl 15:37 0:10 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status_with_redo/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status_with_redo/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1743752394.048717,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00237f500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752394.0487766,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752394.0678859,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002f3e380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752394.0679672,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752394.158516,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005adba40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752394.1585736,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table test.t_100 exists \033[0;36m<<< Run all test success >>>\033[0m + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:39:59 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f8e3d7a-b63e-4e70-96b2-ff6394772043 {"id":"7f8e3d7a-b63e-4e70-96b2-ff6394772043","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752396} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc755e8 7f8e3d7a-b63e-4e70-96b2-ff6394772043 /tidb/cdc/default/default/upstream/7489359462573592486 {"id":7489359462573592486,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f8e3d7a-b63e-4e70-96b2-ff6394772043 {"id":"7f8e3d7a-b63e-4e70-96b2-ff6394772043","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752396} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc755e8 7f8e3d7a-b63e-4e70-96b2-ff6394772043 /tidb/cdc/default/default/upstream/7489359462573592486 {"id":7489359462573592486,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f8e3d7a-b63e-4e70-96b2-ff6394772043 {"id":"7f8e3d7a-b63e-4e70-96b2-ff6394772043","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752396} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc755e8 7f8e3d7a-b63e-4e70-96b2-ff6394772043 /tidb/cdc/default/default/upstream/7489359462573592486 {"id":7489359462573592486,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114227756498945 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.10475.out cli changefeed create --start-ts=457114227756498945 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } table batch_add_table.finish_mark not exists for 1-th check, retry later [Pipeline] // dir Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489359462573592486,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:39:59.649976145+08:00","start_ts":457114227756498945,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114227756498945,"checkpoint_ts":457114227756498945,"checkpoint_time":"2025-04-04 15:39:54.701"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // withEnv [Pipeline] } TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } check diff failed 1-th time, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage + set +x + run_sql 'use test; create table t1(id bigint primary key, a text, b text as ((regexp_replace(a, '\''^[1-9]\d{9,29}$'\'', '\''aaaaa'\''))), c text); insert into t1 (id, a, c) values(1,123456, '\''ab\\\\c'\''); insert into t1 (id, a, c) values(2,1234567890123, '\''ab\\c'\'');' 127.0.0.1 4000 + '[' mysql == mysql ']' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later [Pipeline] } check diff successfully wait process cdc.test exit for 1-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:01 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/585a9c68-c15e-4cf6-864f-232663bd0f5e {"id":"585a9c68-c15e-4cf6-864f-232663bd0f5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752399} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd907d7 585a9c68-c15e-4cf6-864f-232663bd0f5e /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/585a9c68-c15e-4cf6-864f-232663bd0f5e {"id":"585a9c68-c15e-4cf6-864f-232663bd0f5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752399} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd907d7 585a9c68-c15e-4cf6-864f-232663bd0f5e /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/585a9c68-c15e-4cf6-864f-232663bd0f5e {"id":"585a9c68-c15e-4cf6-864f-232663bd0f5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752399} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd907d7 585a9c68-c15e-4cf6-864f-232663bd0f5e /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 5957e1d6-26d3-4fd6-b99e-b5430e140c29 Info: {"upstream_id":7489359484104304026,"namespace":"default","id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:40:02.094338046+08:00","start_ts":457114229677228034,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114229677228034,"checkpoint_ts":457114229677228034,"checkpoint_time":"2025-04-04 15:40:02.028"} table batch_add_table.finish_mark not exists for 2-th check, retry later {"level":"warn","ts":"2025-04-04T15:40:01.803627+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:01.840895+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:01.941726+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} empty select id, val from test.availability3 where id=1 run task successfully delete_owner_key pass wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:02 CST 2025] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table kill_owner_with_ddl.t1 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table test.t1 exists + sleep 10 table batch_add_table.finish_mark not exists for 3-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_resign_owner [Fri Apr 4 15:40:04 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.53045306.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_resign_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_resign_owner.server1 --cluster-id default --addr 127.0.0.1:8300 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table kill_owner_with_ddl.t1 exists wait process 8619 exit for 1-th time... table batch_add_table.finish_mark not exists for 4-th check, retry later wait process 8619 exit for 2-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release check diff failed 4-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8619) - No such process wait process 8619 exit for 3-th time... process 8619 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359529313282680 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8823c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:6321, start at 2025-04-04 15:40:05.178588891 +0800 CST m=+2.562825752 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:05.225 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:05.185 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:05.185 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359529313282680 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8823c0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:6321, start at 2025-04-04 15:40:05.178588891 +0800 CST m=+2.562825752 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:05.225 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:05.185 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:05.185 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359528078020335 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8816c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:6395, start at 2025-04-04 15:40:05.131199895 +0800 CST m=+2.444000868 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:05.141 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:05.132 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:05.132 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... run task successfully [Fri Apr 4 15:40:06 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.87838785.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:07 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004246c60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114229680635909 MinTableBarrierTs:457114230991093766 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63ec4618-716a-49d8-8951-8f6490d8961b {"id":"63ec4618-716a-49d8-8951-8f6490d8961b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752404} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe31b 63ec4618-716a-49d8-8951-8f6490d8961b /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114229680635909,"min-table-barrier-ts":457114230991093766,"admin-job-type":0} /tidb/cdc/default/default/task/position/63ec4618-716a-49d8-8951-8f6490d8961b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004246c60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114229680635909 MinTableBarrierTs:457114230991093766 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63ec4618-716a-49d8-8951-8f6490d8961b {"id":"63ec4618-716a-49d8-8951-8f6490d8961b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752404} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe31b 63ec4618-716a-49d8-8951-8f6490d8961b /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114229680635909,"min-table-barrier-ts":457114230991093766,"admin-job-type":0} /tidb/cdc/default/default/task/position/63ec4618-716a-49d8-8951-8f6490d8961b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004246c60 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114229680635909 MinTableBarrierTs:457114230991093766 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114229680635909, checkpointTs: 457114229680635909, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63ec4618-716a-49d8-8951-8f6490d8961b {"id":"63ec4618-716a-49d8-8951-8f6490d8961b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752404} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe31b 63ec4618-716a-49d8-8951-8f6490d8961b /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114229680635909,"min-table-barrier-ts":457114230991093766,"admin-job-type":0} /tidb/cdc/default/default/task/position/63ec4618-716a-49d8-8951-8f6490d8961b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' {"level":"warn","ts":1743752407.1277947,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00237f500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743752407.2408462,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005adba40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:07.805019+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:07.841697+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} table batch_add_table.finish_mark not exists for 5-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:07.943125+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release "is-owner": true, run task successfully Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) skip some DDLs in tidb v4.0.x [Fri Apr 4 15:40:09 CST 2025] <<<<<< START cdc server in ddl_reentrant case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_reentrant.76287630.out server --log-file /tmp/tidb_cdc_test/ddl_reentrant/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_reentrant/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 5-th time, retry later table batch_add_table.finish_mark not exists for 6-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:09 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00392e120 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231643308035 MinTableBarrierTs:457114231643308035 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231643308035, checkpointTs: 457114231643308035, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8f18c088-c220-40e2-ae95-73ff2b317724 {"id":"8f18c088-c220-40e2-ae95-73ff2b317724","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752407} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd9085f 8f18c088-c220-40e2-ae95-73ff2b317724 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231643308035,"min-table-barrier-ts":457114231643308035,"admin-job-type":0} /tidb/cdc/default/default/task/position/8f18c088-c220-40e2-ae95-73ff2b317724/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00392e120 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231643308035 MinTableBarrierTs:457114231643308035 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231643308035, checkpointTs: 457114231643308035, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8f18c088-c220-40e2-ae95-73ff2b317724 {"id":"8f18c088-c220-40e2-ae95-73ff2b317724","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752407} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd9085f 8f18c088-c220-40e2-ae95-73ff2b317724 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231643308035,"min-table-barrier-ts":457114231643308035,"admin-job-type":0} /tidb/cdc/default/default/task/position/8f18c088-c220-40e2-ae95-73ff2b317724/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00392e120 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231643308035 MinTableBarrierTs:457114231643308035 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231643308035, checkpointTs: 457114231643308035, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8f18c088-c220-40e2-ae95-73ff2b317724 {"id":"8f18c088-c220-40e2-ae95-73ff2b317724","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752407} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd9085f 8f18c088-c220-40e2-ae95-73ff2b317724 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231643308035,"min-table-barrier-ts":457114231643308035,"admin-job-type":0} /tidb/cdc/default/default/task/position/8f18c088-c220-40e2-ae95-73ff2b317724/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:04.354 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:40:04.354 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489358780560304485] [2025/04/04 15:40:04.354 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:40:04.354 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:40:04.355 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:40:04.355 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:40:04.355 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:40:04.355 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:40:04.355 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:40:04.355 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:40:04.356 +08:00] [INFO] [pd_service_discovery.go:1025] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2025/04/04 15:40:04.356 +08:00] [INFO] [pd_service_discovery.go:499] ["[pd] init cluster id"] [cluster-id=7489358780560304485] [2025/04/04 15:40:04.356 +08:00] [INFO] [client.go:532] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:40:04.356 +08:00] [INFO] [tso_client.go:296] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2025/04/04 15:40:04.357 +08:00] [INFO] [client.go:538] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2025/04/04 15:40:04.357 +08:00] [INFO] [tso_dispatcher.go:140] ["[tso] start tso deadline watcher"] [dc-location=global] [2025/04/04 15:40:04.357 +08:00] [INFO] [tso_client.go:132] ["[tso] start tso dispatcher check loop"] [2025/04/04 15:40:04.357 +08:00] [INFO] [tso_dispatcher.go:198] ["[tso] tso dispatcher created"] [dc-location=global] [2025/04/04 15:40:04.357 +08:00] [INFO] [tso_dispatcher.go:476] ["[tso] start tso connection contexts updater"] [dc-location=global] [2025/04/04 15:40:04.357 +08:00] [INFO] [tso_dispatcher.go:723] ["[tso] switching tso rpc concurrency"] [old=0] [new=1] [2025/04/04 15:40:04.357 +08:00] [INFO] [tikv_driver.go:201] ["using API V1."] [2025/04/04 15:40:04.358 +08:00] [INFO] [main.go:180] ["genLock started"] [2025/04/04 15:40:04.359 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=2] [addr=127.0.0.1:20160] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:40:06.358 +08:00] [INFO] [pd.go:441] ["adaptive update ts interval state transition"] [configuredInterval=2s] [prevAdaptiveUpdateInterval=2s] [newAdaptiveUpdateInterval=2s] [requiredStaleness=0s] [prevState=unknown(0)] [newState=normal] owner pid: 5309 owner id 63ec4618-716a-49d8-8951-8f6490d8961b [Fri Apr 4 15:40:10 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.54125414.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_resign_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_resign_owner.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 check diff successfully [2025/04/04 15:40:11.042 +08:00] [INFO] [case.go:115] ["sync updatePKUK take: 11.413756845s"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:12 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b {"id":"fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752409} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe02a4d3 fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b /tidb/cdc/default/default/upstream/7489359529313282680 {"id":7489359529313282680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b {"id":"fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752409} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe02a4d3 fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b /tidb/cdc/default/default/upstream/7489359529313282680 {"id":7489359529313282680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b {"id":"fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752409} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe02a4d3 fcd8f63e-2734-47fe-aa25-11c7ce7c2b8b /tidb/cdc/default/default/upstream/7489359529313282680 {"id":7489359529313282680,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ------------------------------------------ test ddl create database ddl_reentrant, is_reentrant: false restored_sql: CREATE DATABASE `ddl_reentrant` table batch_add_table.finish_mark not exists for 7-th check, retry later check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359553769043860 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d6cc000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:23301, start at 2025-04-04 15:40:10.573350119 +0800 CST m=+2.192273827 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.581 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.547 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.547 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359553769043860 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d6cc000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:23301, start at 2025-04-04 15:40:10.573350119 +0800 CST m=+2.192273827 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.581 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.547 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.547 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359550641358396 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8d6d0000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:23366, start at 2025-04-04 15:40:10.596716199 +0800 CST m=+2.141339008 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:10.636 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:10.598 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:10.598 +0800 All versions after safe point can be accessed. (DO NOT EDIT) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 + run_sql 'SELECT * from test.t1' 127.0.0.1 3306 + check_contains 'b: 123456' + check_contains 'b: aaaaa' + check_contains 'c: ab\\\\c' + check_contains 'c: ab\\c' + stop_tidb_cluster + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:13 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a55290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114232302075914 MinTableBarrierTs:457114232564219914 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63ec4618-716a-49d8-8951-8f6490d8961b {"id":"63ec4618-716a-49d8-8951-8f6490d8961b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752404} /tidb/cdc/default/__cdc_meta__/capture/854a8e42-71f3-4c47-946c-a52ec01f3823 {"id":"854a8e42-71f3-4c47-946c-a52ec01f3823","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe31b 63ec4618-716a-49d8-8951-8f6490d8961b /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe386 854a8e42-71f3-4c47-946c-a52ec01f3823 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114232302075914,"min-table-barrier-ts":457114232564219914,"admin-job-type":0} /tidb/cdc/default/default/task/position/63ec4618-716a-49d8-8951-8f6490d8961b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/854a8e42-71f3-4c47-946c-a52ec01f3823/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a55290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114232302075914 MinTableBarrierTs:457114232564219914 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63ec4618-716a-49d8-8951-8f6490d8961b {"id":"63ec4618-716a-49d8-8951-8f6490d8961b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752404} /tidb/cdc/default/__cdc_meta__/capture/854a8e42-71f3-4c47-946c-a52ec01f3823 {"id":"854a8e42-71f3-4c47-946c-a52ec01f3823","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe31b 63ec4618-716a-49d8-8951-8f6490d8961b /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe386 854a8e42-71f3-4c47-946c-a52ec01f3823 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114232302075914,"min-table-barrier-ts":457114232564219914,"admin-job-type":0} /tidb/cdc/default/default/task/position/63ec4618-716a-49d8-8951-8f6490d8961b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/854a8e42-71f3-4c47-946c-a52ec01f3823/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a55290 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114232302075914 MinTableBarrierTs:457114232564219914 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63ec4618-716a-49d8-8951-8f6490d8961b {"id":"63ec4618-716a-49d8-8951-8f6490d8961b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752404} /tidb/cdc/default/__cdc_meta__/capture/854a8e42-71f3-4c47-946c-a52ec01f3823 {"id":"854a8e42-71f3-4c47-946c-a52ec01f3823","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752410} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe31b 63ec4618-716a-49d8-8951-8f6490d8961b /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe386 854a8e42-71f3-4c47-946c-a52ec01f3823 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114232302075914,"min-table-barrier-ts":457114232564219914,"admin-job-type":0} /tidb/cdc/default/default/task/position/63ec4618-716a-49d8-8951-8f6490d8961b/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/854a8e42-71f3-4c47-946c-a52ec01f3823/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "63ec4618-716a-49d8-8951-8f6490d8961b" | grep id 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:13.8064+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:13.842905+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:13.94437+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table batch_add_table.finish_mark not exists for 8-th check, retry later changefeed is working normally rts: 457114232339824645->457114232366039062 checkpoint: 457114232339824645->457114232720195594 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:40:13.828 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [duration=26.6219ms] [sql="CREATE DATABASE `ddl_reentrant`"] run task successfully resume changefeed 78ff7808-26ad-43e0-b55d-d8dbc7560a95 from 457114232366039062 check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 [2025/04/04 15:40:14.358 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=158] [addr=127.0.0.1:9500] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:40:14.358 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=1] [addr=127.0.0.1:20161] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:40:14.358 +08:00] [INFO] [store_cache.go:532] ["change store resolve state"] [store=5] [addr=127.0.0.1:20162] [from=unresolved] [to=resolved] [liveness-state=reachable] [2025/04/04 15:40:14.370 +08:00] [INFO] [main.go:196] ["genLock done"] [2025/04/04 15:40:14.370 +08:00] [INFO] [pd_service_discovery.go:551] ["[pd] exit member loop due to context canceled"] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_dispatcher.go:264] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_dispatcher.go:490] ["[tso] exit tso connection contexts updater"] [dc-location=global] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_dispatcher.go:158] ["[tso] exit tso deadline watcher"] [dc-location=global] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_dispatcher.go:201] ["[tso] exit tso dispatcher"] [dc-location=global] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_client.go:147] ["[tso] exit tso dispatcher check loop"] [2025/04/04 15:40:14.370 +08:00] [INFO] [resource_manager_client.go:296] ["[resource manager] exit resource token dispatcher"] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_client.go:157] ["[tso] closing tso client"] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_client.go:162] ["[tso] close tso client"] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_client.go:164] ["[tso] tso client is closed"] [2025/04/04 15:40:14.370 +08:00] [INFO] [pd_service_discovery.go:644] ["[pd] close pd service discovery client"] [2025/04/04 15:40:14.370 +08:00] [INFO] [tso_stream.go:359] ["tsoStream.recvLoop ended"] [stream=127.0.0.1:2379-1] [error="rpc error: code = Canceled desc = context canceled"] [errorVerbose="rpc error: code = Canceled desc = context canceled\ngithub.com/tikv/pd/client.(*tsoStream).recvLoop\n\t/go/pkg/mod/github.com/tikv/pd/client@v0.0.0-20241111073742-238d4d79ea31/tso_stream.go:427\nruntime.goexit\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.linux-amd64/src/runtime/asm_amd64.s:1700"] "cluster-id": "default" "id": "854a8e42-71f3-4c47-946c-a52ec01f3823", "cluster-id": "default" run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359564073779471 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8f948000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:9561, start at 2025-04-04 15:40:12.791087762 +0800 CST m=+2.412840436 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:12.799 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:12.804 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:12.804 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359564073779471 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8f948000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:9561, start at 2025-04-04 15:40:12.791087762 +0800 CST m=+2.412840436 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:12.799 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:12.804 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:12.804 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359561473324000 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8fa1c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:9632, start at 2025-04-04 15:40:12.814868641 +0800 CST m=+2.338397996 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:12.823 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:12.857 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:12.857 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359562971040879 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8fb740003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:10117, start at 2025-04-04 15:40:12.925005342 +0800 CST m=+2.446955644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:12.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:12.893 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:12.893 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359562971040879 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8fb740003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:10117, start at 2025-04-04 15:40:12.925005342 +0800 CST m=+2.446955644 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:12.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:12.893 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:12.893 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359560158873041 TiDB Cluster ID. tikv_gc_leader_uuid 657fef8fb780002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-jln6z-32lr7-xj3v0, pid:10191, start at 2025-04-04 15:40:12.924662904 +0800 CST m=+2.367011005 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:12.933 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:12.894 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:12.894 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 1-th time, retry later table batch_add_table.finish_mark not exists for 9-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.24757.out cli tso query --pd=http://127.0.0.1:2379 changefeed is working normally rts: 457114232366039062->457114233244221451 checkpoint: 457114232982339597->457114233244221451 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2025/04/04 15:40:14.806 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114232366039059] [ddl="CREATE DATABASE `ddl_reentrant`"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [error="Error 1007 (HY000): Can't create database 'ddl_reentrant'; database exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t3 (a int primary key, b int) partition by range(a) (partition p0 values less than (1000), partition p1 values less than (2000)), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000)) capture_id: 854a8e42-71f3-4c47-946c-a52ec01f3823 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 2 100 2 0 0 500 0 --:--:-- --:--:-- --:--:-- 666 {}cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 |grep 854a8e42-71f3-4c47-946c-a52ec01f3823 -A1 | grep '"is-owner": true' [Fri Apr 4 15:40:16 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1101411016.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8788) - No such process wait process 8788 exit for 1-th time... process 8788 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... run task successfully [Fri Apr 4 15:40:16 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.89458947.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later ***************** properties ***************** "updateproportion"="0" "scanproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" "operationcount"="0" "workload"="core" "mysql.port"="4000" "recordcount"="20" "readallfields"="true" "mysql.db"="changefeed_auto_stop_1" "insertproportion"="0" "threadcount"="4" "mysql.host"="127.0.0.1" "readproportion"="0" "dotransactions"="false" ********************************************** Run finished, takes 9.484255ms INSERT - Takes(s): 0.0, Count: 19, OPS: 3746.9, Avg(us): 1660, Min(us): 1029, Max(us): 4611, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "requestdistribution"="uniform" "mysql.user"="root" "mysql.db"="changefeed_auto_stop_2" "dotransactions"="false" "workload"="core" "mysql.host"="127.0.0.1" "mysql.port"="4000" "scanproportion"="0" "insertproportion"="0" "readproportion"="0" "operationcount"="0" "updateproportion"="0" "recordcount"="20" "threadcount"="4" "readallfields"="true" ********************************************** Run finished, takes 9.75741ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3652.5, Avg(us): 1707, Min(us): 933, Max(us): 3870, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "mysql.db"="changefeed_auto_stop_3" "readproportion"="0" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "readallfields"="true" "operationcount"="0" "mysql.user"="root" "scanproportion"="0" "insertproportion"="0" "recordcount"="20" "mysql.port"="4000" "threadcount"="4" "updateproportion"="0" "workload"="core" "dotransactions"="false" ********************************************** Run finished, takes 9.4186ms INSERT - Takes(s): 0.0, Count: 20, OPS: 4043.0, Avg(us): 1782, Min(us): 942, Max(us): 4594, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "workload"="core" "recordcount"="20" "mysql.user"="root" "readproportion"="0" "mysql.port"="4000" "operationcount"="0" "insertproportion"="0" "requestdistribution"="uniform" "threadcount"="4" "scanproportion"="0" "readallfields"="true" "mysql.db"="changefeed_auto_stop_4" "mysql.host"="127.0.0.1" "dotransactions"="false" "updateproportion"="0" ********************************************** Run finished, takes 9.693747ms INSERT - Takes(s): 0.0, Count: 19, OPS: 3485.5, Avg(us): 1698, Min(us): 1029, Max(us): 4143, 95th(us): 5000, 99th(us): 5000 [Fri Apr 4 15:40:17 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1153711539.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark not exists for 10-th check, retry later + set +x + tso='457114233404915713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114233404915713 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:17 CST 2025] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.2480024802.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 "is-owner": true, run task successfully test_resign_owner: pass 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 wait process cdc.test exit for 1-th time... run task failed 1-th time, retry later wait process cdc.test exit for 2-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_kill_capture [Fri Apr 4 15:40:19 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.55805582.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 3-th time, retry later table batch_add_table.finish_mark not exists for 11-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:19.808074+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:19.844523+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:19.964655+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:19 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf367a7a-a787-4837-beec-f7cc6851612a {"id":"cf367a7a-a787-4837-beec-f7cc6851612a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe26dcd8 cf367a7a-a787-4837-beec-f7cc6851612a /tidb/cdc/default/default/upstream/7489359564073779471 {"id":7489359564073779471,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf367a7a-a787-4837-beec-f7cc6851612a {"id":"cf367a7a-a787-4837-beec-f7cc6851612a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe26dcd8 cf367a7a-a787-4837-beec-f7cc6851612a /tidb/cdc/default/default/upstream/7489359564073779471 {"id":7489359564073779471,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf367a7a-a787-4837-beec-f7cc6851612a {"id":"cf367a7a-a787-4837-beec-f7cc6851612a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe26dcd8 cf367a7a-a787-4837-beec-f7cc6851612a /tidb/cdc/default/default/upstream/7489359564073779471 {"id":7489359564073779471,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "updateproportion"="0" "workload"="core" "fieldcount"="100" "mysql.port"="4000" "readallfields"="true" "readproportion"="0" "mysql.host"="127.0.0.1" "scanproportion"="0" "mysql.user"="root" "mysql.db"="consistent_replicate_storage_file" "insertproportion"="0" "operationcount"="0" "recordcount"="5000" "dotransactions"="false" "threadcount"="10" "requestdistribution"="uniform" ********************************************** + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:19 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e20480 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/149386ad-3c3f-41d4-b3b9-5a70f185ca1e {"id":"149386ad-3c3f-41d4-b3b9-5a70f185ca1e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd908ec 149386ad-3c3f-41d4-b3b9-5a70f185ca1e /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/149386ad-3c3f-41d4-b3b9-5a70f185ca1e/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e20480 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/149386ad-3c3f-41d4-b3b9-5a70f185ca1e {"id":"149386ad-3c3f-41d4-b3b9-5a70f185ca1e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd908ec 149386ad-3c3f-41d4-b3b9-5a70f185ca1e /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/149386ad-3c3f-41d4-b3b9-5a70f185ca1e/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e20480 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/149386ad-3c3f-41d4-b3b9-5a70f185ca1e {"id":"149386ad-3c3f-41d4-b3b9-5a70f185ca1e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd908ec 149386ad-3c3f-41d4-b3b9-5a70f185ca1e /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/149386ad-3c3f-41d4-b3b9-5a70f185ca1e/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:20 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a108b4d7-8c81-4d5e-bb6c-924bed57df83 {"id":"a108b4d7-8c81-4d5e-bb6c-924bed57df83","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a13 a108b4d7-8c81-4d5e-bb6c-924bed57df83 /tidb/cdc/default/default/upstream/7489359562971040879 {"id":7489359562971040879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a108b4d7-8c81-4d5e-bb6c-924bed57df83 {"id":"a108b4d7-8c81-4d5e-bb6c-924bed57df83","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a13 a108b4d7-8c81-4d5e-bb6c-924bed57df83 /tidb/cdc/default/default/upstream/7489359562971040879 {"id":7489359562971040879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a108b4d7-8c81-4d5e-bb6c-924bed57df83 {"id":"a108b4d7-8c81-4d5e-bb6c-924bed57df83","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a13 a108b4d7-8c81-4d5e-bb6c-924bed57df83 /tidb/cdc/default/default/upstream/7489359562971040879 {"id":7489359562971040879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:40:20 CST 2025] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1160211604.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:20 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/28e66808-490e-4fdf-b82c-b372b36c561b {"id":"28e66808-490e-4fdf-b82c-b372b36c561b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752418} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1734fe 28e66808-490e-4fdf-b82c-b372b36c561b /tidb/cdc/default/default/upstream/7489359553769043860 {"id":7489359553769043860,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/28e66808-490e-4fdf-b82c-b372b36c561b {"id":"28e66808-490e-4fdf-b82c-b372b36c561b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752418} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1734fe 28e66808-490e-4fdf-b82c-b372b36c561b /tidb/cdc/default/default/upstream/7489359553769043860 {"id":7489359553769043860,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/28e66808-490e-4fdf-b82c-b372b36c561b {"id":"28e66808-490e-4fdf-b82c-b372b36c561b","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752418} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe1734fe 28e66808-490e-4fdf-b82c-b372b36c561b /tidb/cdc/default/default/upstream/7489359553769043860 {"id":7489359553769043860,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff failed 4-th time, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:21.791503+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:40:21.791569+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:40:21.828507+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:40:21.828574+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:21.933726+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:40:21.933782+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} lease 223195ffbe1734fe revoked Run finished, takes 1.903496317s INSERT - Takes(s): 1.9, Count: 5000, OPS: 2637.4, Avg(us): 3404, Min(us): 1168, Max(us): 61161, 95th(us): 3000, 99th(us): 49000 table consistent_replicate_storage_file.usertable not exists for 1-th check, retry later table batch_add_table.finish_mark not exists for 12-th check, retry later changefeed is working normally rts: 457114234293059593->457114234830454785 checkpoint: 457114234293059593->457114234830454785 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:40:16.729 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [duration=28.927427ms] [sql="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] run task successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:22 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0044266c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114233612795918 MinTableBarrierTs:457114234923515912 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cce671c-af22-47f8-8f72-77a05e175eec {"id":"7cce671c-af22-47f8-8f72-77a05e175eec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe402 7cce671c-af22-47f8-8f72-77a05e175eec /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114233612795918,"min-table-barrier-ts":457114234923515912,"admin-job-type":0} /tidb/cdc/default/default/task/position/7cce671c-af22-47f8-8f72-77a05e175eec/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0044266c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114233612795918 MinTableBarrierTs:457114234923515912 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cce671c-af22-47f8-8f72-77a05e175eec {"id":"7cce671c-af22-47f8-8f72-77a05e175eec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe402 7cce671c-af22-47f8-8f72-77a05e175eec /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114233612795918,"min-table-barrier-ts":457114234923515912,"admin-job-type":0} /tidb/cdc/default/default/task/position/7cce671c-af22-47f8-8f72-77a05e175eec/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0044266c0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114233612795918 MinTableBarrierTs:457114234923515912 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114233612795918, checkpointTs: 457114233612795918, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cce671c-af22-47f8-8f72-77a05e175eec {"id":"7cce671c-af22-47f8-8f72-77a05e175eec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe402 7cce671c-af22-47f8-8f72-77a05e175eec /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114233612795918,"min-table-barrier-ts":457114234923515912,"admin-job-type":0} /tidb/cdc/default/default/task/position/7cce671c-af22-47f8-8f72-77a05e175eec/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table test.t2 not exists for 1-th check, retry later resume changefeed 78ff7808-26ad-43e0-b55d-d8dbc7560a95 from 457114233493520398 check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table capture_session_done_during_task.t exists check diff failed 5-th time, retry later check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:23 GMT < Content-Length: 1261 < Content-Type: text/plain; charset=utf-8 < { [1261 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a108b4d7-8c81-4d5e-bb6c-924bed57df83 {"id":"a108b4d7-8c81-4d5e-bb6c-924bed57df83","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/capture/ce9d8c42-cdec-40a8-b501-a64e152e097a {"id":"ce9d8c42-cdec-40a8-b501-a64e152e097a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a13 a108b4d7-8c81-4d5e-bb6c-924bed57df83 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a52 ce9d8c42-cdec-40a8-b501-a64e152e097a /tidb/cdc/default/default/upstream/7489359562971040879 {"id":7489359562971040879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a108b4d7-8c81-4d5e-bb6c-924bed57df83 {"id":"a108b4d7-8c81-4d5e-bb6c-924bed57df83","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/capture/ce9d8c42-cdec-40a8-b501-a64e152e097a {"id":"ce9d8c42-cdec-40a8-b501-a64e152e097a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a13 a108b4d7-8c81-4d5e-bb6c-924bed57df83 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a52 ce9d8c42-cdec-40a8-b501-a64e152e097a /tidb/cdc/default/default/upstream/7489359562971040879 {"id":7489359562971040879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a108b4d7-8c81-4d5e-bb6c-924bed57df83 {"id":"a108b4d7-8c81-4d5e-bb6c-924bed57df83","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752417} /tidb/cdc/default/__cdc_meta__/capture/ce9d8c42-cdec-40a8-b501-a64e152e097a {"id":"ce9d8c42-cdec-40a8-b501-a64e152e097a","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752420} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a13 a108b4d7-8c81-4d5e-bb6c-924bed57df83 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe215a52 ce9d8c42-cdec-40a8-b501-a64e152e097a /tidb/cdc/default/default/upstream/7489359562971040879 {"id":7489359562971040879,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 090b90f3-e059-43f6-891a-3694d5912361 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=090b90f3-e059-43f6-891a-3694d5912361 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 090b90f3-e059-43f6-891a-3694d5912361 -s + info='{ "upstream_id": 7489359562971040879, "namespace": "default", "id": "090b90f3-e059-43f6-891a-3694d5912361", "state": "normal", "checkpoint_tso": 457114233587105797, "checkpoint_time": "2025-04-04 15:40:16.943", "error": null }' + echo '{ "upstream_id": 7489359562971040879, "namespace": "default", "id": "090b90f3-e059-43f6-891a-3694d5912361", "state": "normal", "checkpoint_tso": 457114233587105797, "checkpoint_time": "2025-04-04 15:40:16.943", "error": null }' { "upstream_id": 7489359562971040879, "namespace": "default", "id": "090b90f3-e059-43f6-891a-3694d5912361", "state": "normal", "checkpoint_tso": 457114233587105797, "checkpoint_time": "2025-04-04 15:40:16.943", "error": null } ++ echo '{' '"upstream_id":' 7489359562971040879, '"namespace":' '"default",' '"id":' '"090b90f3-e059-43f6-891a-3694d5912361",' '"state":' '"normal",' '"checkpoint_tso":' 457114233587105797, '"checkpoint_time":' '"2025-04-04' '15:40:16.943",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359562971040879, '"namespace":' '"default",' '"id":' '"090b90f3-e059-43f6-891a-3694d5912361",' '"state":' '"normal",' '"checkpoint_tso":' 457114233587105797, '"checkpoint_time":' '"2025-04-04' '15:40:16.943",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later "is-owner": true, run task successfully table batch_add_table.finish_mark not exists for 13-th check, retry later table test.t2 not exists for 2-th check, retry later table consistent_replicate_storage_file.usertable exists table consistent_replicate_storage_file.check1 not exists for 1-th check, retry later run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 owner pid: 5585 owner id 7cce671c-af22-47f8-8f72-77a05e175eec table test.availability1 exists nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later pass test: get status pass test: health [2025/04/04 15:40:18.727 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info=null] [2025/04/04 15:40:21.330 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info="{\"upstream_id\":7489358659000709430,\"namespace\":\"test\",\"id\":\"changefeed-test-v2-black-hole-1\",\"sink_uri\":\"blackhole:\",\"create_time\":\"2025-04-04T15:40:18.670676724+08:00\",\"start_ts\":457114234007584773,\"config\":{\"memory_quota\":1073741824,\"case_sensitive\":false,\"force_replicate\":false,\"ignore_ineligible_table\":false,\"check_gc_safe_point\":true,\"enable_sync_point\":false,\"bdr_mode\":false,\"sync_point_interval\":600000000000,\"sync_point_retention\":86400000000000,\"filter\":{\"rules\":[\"*.*\"]},\"mounter\":{\"worker_num\":16},\"sink\":{\"csv\":{\"delimiter\":\",\",\"quote\":\"\\\"\",\"null\":\"\\\\N\",\"include_commit_ts\":false},\"transaction_atomicity\":\"\",\"encoder_concurrency\":32,\"terminator\":\"\\r\\n\",\"date_separator\":\"day\",\"enable_partition_separator\":true,\"content_compatible\":false,\"send_bootstrap_interval_in_sec\":120,\"send_bootstrap_in_msg_count\":10000,\"send_bootstrap_to_all_partition\":true,\"debezium_disable_schema\":false,\"debezium\":{\"output_old_value\":true},\"open\":{\"output_old_value\":true}},\"consistent\":{\"level\":\"none\",\"max_log_size\":64,\"flush_interval\":2000,\"meta_flush_interval\":200,\"encoding_worker_num\":16,\"flush_worker_num\":8,\"storage\":\"\",\"use_file_backend\":false},\"scheduler\":{\"enable_table_across_nodes\":false,\"region_threshold\":100000,\"write_key_threshold\":0},\"integrity\":{\"integrity_check_level\":\"none\",\"corruption_handle_level\":\"warn\"}},\"state\":\"normal\",\"creator_version\":\"v9.0.0-beta.1-10-ga306f4894\",\"resolved_ts\":0,\"checkpoint_ts\":457114234413907971,\"checkpoint_time\":\"2025-04-04 15:40:20.097\"}"] check diff successfully wait process cdc.test exit for 1-th time... check diff failed 2-th time, retry later + start_tidb_cluster --workdir /tmp/tidb_cdc_test/sql_mode The 1 times to try to start tidb cluster... 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:25.808972+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:25.845645+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:25.965485+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table batch_add_table.finish_mark not exists for 14-th check, retry later [2025/04/04 15:40:26.259 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info="{\"upstream_id\":7489358659000709430,\"namespace\":\"test\",\"id\":\"changefeed-test-v2-black-hole-1\",\"sink_uri\":\"blackhole:\",\"create_time\":\"2025-04-04T15:40:18.670676724+08:00\",\"start_ts\":457114234007584773,\"admin_job_type\":1,\"config\":{\"memory_quota\":1123450,\"case_sensitive\":false,\"force_replicate\":false,\"ignore_ineligible_table\":false,\"check_gc_safe_point\":false,\"enable_sync_point\":false,\"bdr_mode\":false,\"sync_point_interval\":600000000000,\"sync_point_retention\":86400000000000,\"filter\":{\"rules\":[\"a.d\",\"b.x\"],\"ignore_txn_start_ts\":[1,2,3],\"event_filters\":[{\"matcher\":[\"test.worker\"],\"ignore_event\":[\"update\"],\"ignore_sql\":[\"^drop\",\"add column\"],\"ignore_insert_value_expr\":\"id \\u003e= 100\",\"ignore_update_new_value_expr\":\"gender = 'male'\",\"ignore_update_old_value_expr\":\"age \\u003c 18\",\"ignore_delete_value_expr\":\"id \\u003e 100\"}]},\"mounter\":{\"worker_num\":17},\"sink\":{\"protocol\":\"arvo\",\"csv\":{\"delimiter\":\",\",\"quote\":\"\\\"\",\"null\":\"\\\\N\",\"include_commit_ts\":false},\"column_selectors\":[{\"matcher\":[\"a.b\"],\"columns\":[\"c\"]}],\"transaction_atomicity\":\"table\",\"encoder_concurrency\":32,\"terminator\":\"a\",\"date_separator\":\"day\",\"enable_partition_separator\":true,\"content_compatible\":true,\"send_bootstrap_interval_in_sec\":120,\"send_bootstrap_in_msg_count\":10000,\"send_bootstrap_to_all_partition\":true,\"debezium_disable_schema\":true,\"debezium\":{\"output_old_value\":true},\"open\":{\"output_old_value\":true}},\"consistent\":{\"level\":\"none\",\"max_log_size\":64,\"flush_interval\":2000,\"meta_flush_interval\":200,\"encoding_worker_num\":31,\"flush_worker_num\":18,\"storage\":\"\",\"use_file_backend\":false},\"scheduler\":{\"enable_table_across_nodes\":false,\"region_threshold\":13,\"write_key_threshold\":0},\"integrity\":{\"integrity_check_level\":\"none\",\"corruption_handle_level\":\"warn\"}},\"state\":\"stopped\",\"creator_version\":\"v9.0.0-beta.1-10-ga306f4894\",\"resolved_ts\":457114234413907971,\"checkpoint_ts\":457114234413907971,\"checkpoint_time\":\"2025-04-04 15:40:20.097\"}"] wait process cdc.test exit for 2-th time... table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later table test.t2 not exists for 3-th check, retry later table consistent_replicate_storage_file.check1 not exists for 2-th check, retry later check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:26 CST 2025] <<<<<< run test case changefeed_pause_resume success! >>>>>> check diff successfully check diff failed 1-th time, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully [Fri Apr 4 15:40:27 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.57115713.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark not exists for 15-th check, retry later table changefeed_auto_stop_2.usertable not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table test.t2 not exists for 4-th check, retry later table consistent_replicate_storage_file.check1 not exists for 3-th check, retry later changefeed is working normally rts: 457114236141174788->457114236403318787 checkpoint: 457114236141174787->457114236403318787 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2025/04/04 15:40:24.801 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114233493520394] [ddl="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [error="Error 1050 (42S01): Table 'ddl_reentrant.t3' already exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t4 (a int primary key, b int), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT) check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 pass test: changefeed apis [2025/04/04 15:40:28.777 +08:00] [INFO] [cases.go:432] ["check changefeed failed"] [time=0] [info=null] 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1743752429.049445,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00237f500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752429.049503,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752429.0698311,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002f3e380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752429.0698893,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752429.1592584,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc005adba40/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752429.1593232,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359616171577750 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9de8c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:7357, start at 2025-04-04 15:40:27.433244285 +0800 CST m=+4.749989430 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:27.441 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:27.427 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:27.427 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359616171577750 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9de8c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:7357, start at 2025-04-04 15:40:27.433244285 +0800 CST m=+4.749989430 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:27.441 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:27.427 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:27.427 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359611522287620 TiDB Cluster ID. tikv_gc_leader_uuid 657fef9dd1c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-txjnm-sgfcp-t4l5j, pid:7426, start at 2025-04-04 15:40:27.356122761 +0800 CST m=+4.574174522 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:27.363 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:27.335 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:27.335 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table batch_add_table.finish_mark not exists for 16-th check, retry later table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process 8950 exit for 1-th time... wait process 8950 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8950) - No such process wait process 8950 exit for 3-th time... process 8950 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:40:29 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.91019103.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_file.check1 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037fd200 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114237033775105 MinTableBarrierTs:457114237033775105 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a544294-31c9-4565-9951-2322f3f01f38 {"id":"4a544294-31c9-4565-9951-2322f3f01f38","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/capture/7cce671c-af22-47f8-8f72-77a05e175eec {"id":"7cce671c-af22-47f8-8f72-77a05e175eec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe402 7cce671c-af22-47f8-8f72-77a05e175eec /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe464 4a544294-31c9-4565-9951-2322f3f01f38 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114237033775105,"min-table-barrier-ts":457114237033775105,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a544294-31c9-4565-9951-2322f3f01f38/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7cce671c-af22-47f8-8f72-77a05e175eec/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037fd200 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114237033775105 MinTableBarrierTs:457114237033775105 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a544294-31c9-4565-9951-2322f3f01f38 {"id":"4a544294-31c9-4565-9951-2322f3f01f38","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/capture/7cce671c-af22-47f8-8f72-77a05e175eec {"id":"7cce671c-af22-47f8-8f72-77a05e175eec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe402 7cce671c-af22-47f8-8f72-77a05e175eec /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe464 4a544294-31c9-4565-9951-2322f3f01f38 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114237033775105,"min-table-barrier-ts":457114237033775105,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a544294-31c9-4565-9951-2322f3f01f38/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7cce671c-af22-47f8-8f72-77a05e175eec/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' changefeed is working normally rts: 457114236403318787->457114236927344645 checkpoint: 457114236403318787->457114236625879055 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:40:29.830 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [duration=30.416536ms] [sql="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] run task successfully + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037fd200 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114237033775105 MinTableBarrierTs:457114237033775105 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a544294-31c9-4565-9951-2322f3f01f38 {"id":"4a544294-31c9-4565-9951-2322f3f01f38","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752427} /tidb/cdc/default/__cdc_meta__/capture/7cce671c-af22-47f8-8f72-77a05e175eec {"id":"7cce671c-af22-47f8-8f72-77a05e175eec","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752419} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe402 7cce671c-af22-47f8-8f72-77a05e175eec /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe464 4a544294-31c9-4565-9951-2322f3f01f38 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114237033775105,"min-table-barrier-ts":457114237033775105,"admin-job-type":0} /tidb/cdc/default/default/task/position/4a544294-31c9-4565-9951-2322f3f01f38/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/7cce671c-af22-47f8-8f72-77a05e175eec/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "7cce671c-af22-47f8-8f72-77a05e175eec" | grep id wait process cdc.test exit for 2-th time... table test.t2 not exists for 5-th check, retry later resume changefeed 78ff7808-26ad-43e0-b55d-d8dbc7560a95 from 457114236625879055 check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 [Fri Apr 4 15:40:31 CST 2025] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.87188720.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:31 CST 2025] <<<<<< run test case capture_session_done_during_task success! >>>>>> pass test: delete changefeed apis pass test: capture apis pass test: processor apis 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:31.810334+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:31.846666+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:40:31.965956+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later "cluster-id": "default" "id": "4a544294-31c9-4565-9951-2322f3f01f38", "cluster-id": "default" run task successfully table batch_add_table.finish_mark not exists for 17-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:32 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [4088 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00382a750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3 {"id":"5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752430} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd9097a 5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00382a750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3 {"id":"5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752430} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd9097a 5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00382a750 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3 {"id":"5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752430} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd9097a 5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/5aaf349e-5a39-4ffd-908d-d0fb6b70e5b3/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table test.t2 not exists for 6-th check, retry later table consistent_replicate_storage_file.check1 not exists for 5-th check, retry later run task failed 1-th time, retry later [2025/04/04 15:40:32.185 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:54' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.185 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:55' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.189 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:59' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.226 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:56' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.226 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:57' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.226 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:55' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.230 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:58' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.234 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:56' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.238 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:57' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.240 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:58' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.244 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:39' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.247 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:50' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.251 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:53' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.257 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:51' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.258 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:50:52' for key 'itest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.278 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:51:39' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.327 +08:00] [ERROR] [job.go:75] ["Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.341 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:52:00' for key 'ptest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.356 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:51:40' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.369 +08:00] [ERROR] [job.go:75] ["Error 1062 (23000): Duplicate entry '15:52:01' for key 'ntest.d'"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] [2025/04/04 15:40:32.772 +08:00] [ERROR] [job.go:75] ["Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] check_capture_count http://127.0.0.1:2379 1 run task successfully [2025/04/04 15:40:33.269 +08:00] [ERROR] [job.go:75] ["Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] nonempty select id, val from test.availability1 where id=2 and val=2 [2025/04/04 15:40:33.332 +08:00] [ERROR] [job.go:75] ["Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction"] [stack="github.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.execSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:75\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doSqls\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:63\ngithub.com/pingcap/tiflow/tests/integration_tests/cdc/dailytest.doJob\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cdc/dailytest/job.go:90"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table batch_add_table.finish_mark not exists for 18-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:34 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f7f15090-a5d3-4556-a11e-4110b26c162a {"id":"f7f15090-a5d3-4556-a11e-4110b26c162a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752431} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4d1bdb f7f15090-a5d3-4556-a11e-4110b26c162a /tidb/cdc/default/default/upstream/7489359616171577750 {"id":7489359616171577750,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f7f15090-a5d3-4556-a11e-4110b26c162a {"id":"f7f15090-a5d3-4556-a11e-4110b26c162a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752431} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4d1bdb f7f15090-a5d3-4556-a11e-4110b26c162a /tidb/cdc/default/default/upstream/7489359616171577750 {"id":7489359616171577750,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f7f15090-a5d3-4556-a11e-4110b26c162a {"id":"f7f15090-a5d3-4556-a11e-4110b26c162a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752431} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe4d1bdb f7f15090-a5d3-4556-a11e-4110b26c162a /tidb/cdc/default/default/upstream/7489359616171577750 {"id":7489359616171577750,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later table test.t2 not exists for 7-th check, retry later check_changefeed_status 127.0.0.1:8300 d68440b6-425b-4b8a-9f6f-22736905ec25 normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=d68440b6-425b-4b8a-9f6f-22736905ec25 + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/d68440b6-425b-4b8a-9f6f-22736905ec25/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 905 0 --:--:-- --:--:-- --:--:-- 905 + info='{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' + echo '{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' {"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034} ++ echo '{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 d68440b6-425b-4b8a-9f6f-22736905ec25 normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=d68440b6-425b-4b8a-9f6f-22736905ec25 + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/d68440b6-425b-4b8a-9f6f-22736905ec25/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 table consistent_replicate_storage_file.check1 not exists for 6-th check, retry later pass test: owner apis pass test: set log level Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1116 0 --:--:-- --:--:-- --:--:-- 1116 + info='{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' + echo '{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' {"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034} ++ echo '{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":457114238131372034,"checkpoint_ts":457114238131372034}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + cleanup_process cdc.test nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 table batch_add_table.finish_mark not exists for 19-th check, retry later check diff successfully table test.t2 not exists for 8-th check, retry later wait process cdc.test exit for 2-th time... table consistent_replicate_storage_file.check1 not exists for 7-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit ++ date + echo '[Fri Apr 4 15:40:36 CST 2025] <<<<<< run test case api_v2 success! >>>>>>' [Fri Apr 4 15:40:36 CST 2025] <<<<<< run test case api_v2 success! >>>>>> + stop_tidb_cluster =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cli_with_auth/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table sink_hang.t1 exists table sink_hang.t2 not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359467753540872 TiDB Cluster ID. tikv_gc_leader_uuid 657fef79994000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9146, start at 2025-04-04 15:39:50.273178944 +0800 CST m=+2.246077329 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.281 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) changefeed is working normally rts: 457114238238064645->457114238500208645 checkpoint: 457114238238064644->457114238500208644 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2025/04/04 15:40:32.800 +08:00] [INFO] [mysql_ddl_sink.go:150] ["Execute DDL failed, but error can be ignored"] [startTs=457114236625879052] [ddl="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [error="Error 1050 (42S01): Table 'ddl_reentrant.t4' already exists"] run task successfully ------------------------------------------ test ddl alter table ddl_reentrant.t3 exchange partition p0 with table ddl_reentrant.t4 without validation, is_reentrant: true restored_sql: ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` WITHOUT VALIDATION check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 2 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test {"level":"warn","ts":"2025-04-04T15:40:37.811863+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001c80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:37.847526+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001f00c40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:40:37.967135+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00088d880/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table batch_add_table.finish_mark not exists for 20-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:37 CST 2025] <<<<<< run test case changefeed_auto_stop success! >>>>>> wait process cdc.test exit for 1-th time... table test.t2 not exists for 9-th check, retry later table consistent_replicate_storage_file.check1 not exists for 8-th check, retry later wait process cdc.test exit for 2-th time... changefeed is working normally rts: 457114238762614788->457114239024758790 checkpoint: 457114238762614788->457114239024758789 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:40:38.003 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [duration=47.548036ms] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` WITHOUT VALIDATION"] run task successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster table sink_hang.t2 exists check diff failed 1-th time, retry later resume changefeed 78ff7808-26ad-43e0-b55d-d8dbc7560a95 from 457114238788829212 check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 ____________________________________ *************************** 1. row *************************** primary_ts: 457114174095360000 secondary_ts: 457114175884492836 *************************** 2. row *************************** primary_ts: 457114181959680000 secondary_ts: 457114182215270401 *************************** 3. row *************************** primary_ts: 457114189824000000 secondary_ts: 457114190079852545 *************************** 4. row *************************** primary_ts: 457114197688320000 secondary_ts: 457114197944172545 *************************** 5. row *************************** primary_ts: 457114205552640000 secondary_ts: 457114205834444801 *************************** 6. row *************************** primary_ts: 457114213416960000 secondary_ts: 457114213699026945 *************************** 7. row *************************** primary_ts: 457114221281280000 secondary_ts: 457114221563346945 *************************** 8. row *************************** primary_ts: 457114229145600000 secondary_ts: 457114229453619201 skip invalid syncpoint primary_ts: 457114174095360000, first_ddl_ts: 457114211587719180 skip invalid syncpoint primary_ts: 457114181959680000, first_ddl_ts: 457114211587719180 skip invalid syncpoint primary_ts: 457114189824000000, first_ddl_ts: 457114211587719180 skip invalid syncpoint primary_ts: 457114197688320000, first_ddl_ts: 457114211587719180 skip invalid syncpoint primary_ts: 457114205552640000, first_ddl_ts: 457114211587719180 check diff successfully check diff successfully check diff successfully check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:40:33 CST 2025] <<<<<< run test case syncpoint success! >>>>>> nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later table batch_add_table.finish_mark not exists for 21-th check, retry later [2025/04/04 15:40:39.548 +08:00] [WARN] [diff.go:557] ["checksum is not equal"] [table=`test`.`itest`] [where="((TRUE) AND TRUE)"] [sourceChecksum=2633806901] [targetChecksum=1467184114] [getSourceChecksumCost=5.973207ms] [getTargetChecksumCost=2.602416ms] [2025/04/04 15:40:39.560 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:31, a: 0, b: 0, }"] [row2="{ a: 0, b: 0, c: fBr8J1vQ, zFzkK: IsNull, d: 15:40:31, }"] [2025/04/04 15:40:39.633 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:53, a: 1, b: 1, c: Nw9p, }"] [row2="{ a: 1, b: 1, c: K8Qn71, zFzkK: IsNull, d: 15:43:53, }"] [2025/04/04 15:40:39.734 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:35, a: 4, b: 4, }"] [row2="{ a: 4, b: 4, c: kdzUt4d, zFzkK: IsNull, d: 15:40:35, }"] [2025/04/04 15:40:39.834 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 5, b: 5, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:51, }"] [row2="{ a: 5, b: 5, c: G, zFzkK: IsNull, d: 15:43:51, }"] [2025/04/04 15:40:39.934 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:37, a: 6, b: 6, }"] [row2="{ d: 15:40:37, a: 6, b: 6, c: pzb, zFzkK: IsNull, }"] [2025/04/04 15:40:40.034 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 7, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:38, a: 7, }"] [row2="{ a: 7, b: 7, c: y, zFzkK: IsNull, d: 15:40:38, }"] start tidb cluster in /tmp/tidb_cdc_test/cli_with_auth Starting Upstream PD... [2025/04/04 15:40:40.134 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:40:39, a: 8, b: 8, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 8, b: 8, c: 4AGL2jM, zFzkK: IsNull, d: 15:40:39, }"] [2025/04/04 15:40:40.235 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:41, a: 10, b: 10, }"] [row2="{ d: 15:40:41, a: 10, b: 10, c: Jj4k, zFzkK: IsNull, }"] Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:40:40.335 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 12, b: 12, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:57:00, }"] [row2="{ a: 12, b: 12, c: E4, zFzkK: IsNull, d: 15:40:43, }"] [2025/04/04 15:40:40.435 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:40:45, a: 14, b: 14, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:40:45, a: 14, b: 14, c: SRky7k, }"] [2025/04/04 15:40:40.535 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 15, b: 15, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:46, }"] [row2="{ c: CEWr7QWT5J, zFzkK: IsNull, d: 15:40:46, a: 15, b: 15, }"] [2025/04/04 15:40:40.635 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 17, b: 17, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:48, }"] [row2="{ c: dwrzl6b, zFzkK: IsNull, d: 15:40:48, a: 17, b: 17, }"] [2025/04/04 15:40:40.736 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 18, b: 18, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:49, }"] [row2="{ a: 18, b: 18, c: 8, zFzkK: IsNull, d: 15:40:49, }"] table test.t2 not exists for 10-th check, retry later [2025/04/04 15:40:40.836 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 20, b: 20, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:51, }"] [row2="{ a: 20, b: 20, c: jm, zFzkK: IsNull, d: 15:40:51, }"] [2025/04/04 15:40:40.936 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 21, b: 21, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:52, }"] [row2="{ c: DmLV, zFzkK: IsNull, d: 15:40:52, a: 21, b: 21, }"] [2025/04/04 15:40:41.036 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:54, a: 23, b: 23, }"] [row2="{ b: 23, c: stFc, zFzkK: IsNull, d: 15:40:54, a: 23, }"] table consistent_replicate_storage_file.check1 not exists for 9-th check, retry later [2025/04/04 15:40:41.136 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:56, a: 24, b: 24, }"] [row2="{ a: 24, b: 24, c: WucHr, zFzkK: IsNull, d: 15:43:56, }"] [2025/04/04 15:40:41.237 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:56, a: 25, b: 25, }"] [row2="{ a: 25, b: 25, c: bO4OCP, zFzkK: IsNull, d: 15:40:56, }"] check diff failed 2-th time, retry later run task failed 1-th time, retry later [2025/04/04 15:40:41.337 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:58, a: 27, b: 27, }"] [row2="{ a: 27, b: 27, c: 49UD, zFzkK: IsNull, d: 15:40:58, }"] [2025/04/04 15:40:41.437 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:40:59, a: 28, b: 28, c: Nw9p, }"] [row2="{ a: 28, b: 28, c: wb, zFzkK: IsNull, d: 15:40:59, }"] [2025/04/04 15:40:41.537 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 29, b: 29, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:00, }"] [row2="{ a: 29, b: 29, c: g4GFo9h8, zFzkK: IsNull, d: 15:41:00, }"] [2025/04/04 15:40:41.637 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 30, b: 30, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:01, }"] [row2="{ a: 30, b: 30, c: s, zFzkK: IsNull, d: 15:41:01, }"] [2025/04/04 15:40:41.738 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 31, b: 31, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:02, }"] [row2="{ zFzkK: IsNull, d: 15:41:02, a: 31, b: 31, c: ewNJ7rgnGV, }"] [2025/04/04 15:40:41.838 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 32, b: 32, c: qOz, zFzkK: IsNull, d: 15:41:03, }"] [2025/04/04 15:40:41.938 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 33, b: 33, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:04, }"] [row2="{ c: d5Ou5, zFzkK: IsNull, d: 15:41:04, a: 33, b: 33, }"] [2025/04/04 15:40:42.038 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:07, a: 36, b: 36, }"] [row2="{ b: 36, c: fZTKq0C, zFzkK: IsNull, d: 15:41:07, a: 36, }"] [2025/04/04 15:40:42.138 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:08, a: 37, b: 37, c: Nw9p, }"] [row2="{ a: 37, b: 37, c: V, zFzkK: IsNull, d: 15:41:08, }"] [2025/04/04 15:40:42.239 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 39, b: 39, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:10, }"] [row2="{ c: bZ, zFzkK: IsNull, d: 15:41:10, a: 39, b: 39, }"] [2025/04/04 15:40:42.339 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 40, b: 40, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:11, }"] [row2="{ a: 40, b: 40, c: zdDeX4C0bL, zFzkK: IsNull, d: 15:41:11, }"] table batch_add_table.finish_mark not exists for 22-th check, retry later [2025/04/04 15:40:42.439 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:13, a: 42, b: 42, c: Nw9p, }"] [row2="{ a: 42, b: 42, c: LASaYv, zFzkK: IsNull, d: 15:41:13, }"] [2025/04/04 15:40:42.539 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 43, b: 43, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:14, }"] [row2="{ d: 15:41:14, a: 43, b: 43, c: Hg, zFzkK: IsNull, }"] [2025/04/04 15:40:42.639 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 44, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:10, a: 44, }"] [row2="{ c: e4n384dPef, zFzkK: IsNull, d: 15:46:10, a: 44, b: 44, }"] [2025/04/04 15:40:42.740 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:41:16, a: 45, b: 45, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: SmSjV, zFzkK: IsNull, d: 15:41:16, a: 45, b: 45, }"] [2025/04/04 15:40:42.840 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 46, b: 47, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:18, }"] [row2="{ zFzkK: IsNull, d: 15:41:18, a: 46, b: 47, c: oKH, }"] table test.t2 not exists for 11-th check, retry later wait process 9106 exit for 1-th time... wait process 9106 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (9106) - No such process wait process 9106 exit for 3-th time... process 9106 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 4 15:40:42 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.92539255.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_file.check1 not exists for 10-th check, retry later [2025/04/04 15:40:42.940 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 47, b: 46, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:17, }"] [row2="{ zFzkK: IsNull, d: 15:41:17, a: 47, b: 46, c: D5QNVsV7q4, }"] [2025/04/04 15:40:43.040 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 48, b: 48, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:19, }"] [row2="{ c: BSEtSPYNL, zFzkK: IsNull, d: 15:41:19, a: 48, b: 48, }"] check_ts_forward 78ff7808-26ad-43e0-b55d-d8dbc7560a95 [2025/04/04 15:40:43.140 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 50, b: 50, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:21, }"] [row2="{ a: 50, b: 50, c: SPtJsSBj, zFzkK: IsNull, d: 15:41:21, }"] [2025/04/04 15:40:43.241 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:41:22, a: 51, b: 51, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 51, b: 51, c: bhrecc, zFzkK: IsNull, d: 15:41:22, }"] [2025/04/04 15:40:43.341 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 53, b: 53, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:24, }"] [row2="{ c: vRD7Ttvj, zFzkK: IsNull, d: 15:41:24, a: 53, b: 53, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:40:43.441 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 54, b: 54, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:25, }"] [row2="{ a: 54, b: 54, c: O, zFzkK: IsNull, d: 15:41:25, }"] [2025/04/04 15:40:43.541 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:41:26, a: 55, b: 55, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 55, b: 55, c: C6YFNdjmL8, zFzkK: IsNull, d: 15:41:26, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:40:43.641 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:41:27, a: 56, b: 56, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: grTVSPfE, zFzkK: IsNull, d: 15:41:27, a: 56, b: 56, }"] [2025/04/04 15:40:43.742 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 57, b: 57, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:11, }"] [row2="{ a: 57, b: 57, c: H, zFzkK: IsNull, d: 15:47:11, }"] [2025/04/04 15:40:43.842 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 59, b: 59, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:30, }"] [row2="{ d: 15:41:30, a: 59, b: 59, c: FMHrvMuAyE, zFzkK: IsNull, }"] check diff successfully wait process cdc.test exit for 1-th time... [2025/04/04 15:40:43.942 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:31, a: 60, b: 60, }"] [row2="{ c: EcZY6N, zFzkK: IsNull, d: 15:41:31, a: 60, b: 60, }"] [2025/04/04 15:40:44.042 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:32, a: 61, b: 61, c: Nw9p, }"] [row2="{ a: 61, b: 61, c: mtX3wgZ, zFzkK: IsNull, d: 15:41:32, }"] table batch_add_table.finish_mark not exists for 23-th check, retry later [2025/04/04 15:40:44.142 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:33, a: 62, b: 62, c: Nw9p, }"] [row2="{ a: 62, b: 62, c: QDQLFvwbj, zFzkK: IsNull, d: 15:41:33, }"] [2025/04/04 15:40:44.243 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 63, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:34, a: 63, }"] [row2="{ a: 63, b: 63, c: HfOyewFE, zFzkK: IsNull, d: 15:41:34, }"] [2025/04/04 15:40:44.343 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 64, b: 64, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:35, }"] [row2="{ d: 15:41:35, a: 64, b: 64, c: Udgw, zFzkK: IsNull, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:40:44.443 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 65, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:36, a: 65, }"] [row2="{ b: 65, c: 8ZB6Yc2, zFzkK: IsNull, d: 15:41:36, a: 65, }"] [2025/04/04 15:40:44.543 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:39, a: 68, b: 68, c: Nw9p, }"] [row2="{ d: 15:41:39, a: 68, b: 68, c: SGB, zFzkK: IsNull, }"] table test.t2 not exists for 12-th check, retry later [2025/04/04 15:40:44.644 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ b: 70, c: eV, zFzkK: IsNull, d: 15:41:41, a: 70, }"] [2025/04/04 15:40:44.744 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:42, a: 71, b: 71, }"] [row2="{ zFzkK: IsNull, d: 15:41:42, a: 71, b: 71, c: oh9ba, }"] [2025/04/04 15:40:44.844 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 72, b: 72, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:52, }"] [row2="{ d: 15:43:52, a: 72, b: 72, c: Kq8BlgU, zFzkK: IsNull, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint_check_ts/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table consistent_replicate_storage_file.check1 not exists for 11-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:44 CST 2025] <<<<<< run test case sink_hang success! >>>>>> [2025/04/04 15:40:44.944 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 73, b: 73, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:44, }"] [row2="{ a: 73, b: 73, c: uHlgG, zFzkK: IsNull, d: 15:41:44, }"] [2025/04/04 15:40:45.044 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ d: 15:41:45, a: 74, b: 74, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ d: 15:41:45, a: 74, b: 74, c: olVKfhn, zFzkK: IsNull, }"] changefeed is working normally rts: 457114240335216645->457114240859504646 checkpoint: 457114240335216645->457114240597622790 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2025/04/04 15:40:39.913 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=default] [changefeed=78ff7808-26ad-43e0-b55d-d8dbc7560a95] [duration=46.899597ms] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` WITHOUT VALIDATION"] run task successfully [2025/04/04 15:40:45.145 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ d: 15:41:46, a: 75, b: 75, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 75, b: 75, c: WyU7mxGs, zFzkK: IsNull, d: 15:41:46, }"] [2025/04/04 15:40:45.245 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:47, a: 76, b: 76, }"] [row2="{ a: 76, b: 76, c: cUH0TQuc, zFzkK: IsNull, d: 15:41:47, }"] [2025/04/04 15:40:45.345 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:41:48, a: 77, b: 77, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:41:48, a: 77, b: 77, c: ep, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:45.445 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:49, a: 78, b: 78, c: Nw9p, }"] [row2="{ a: 78, b: 78, c: zQG9gSGVU, zFzkK: IsNull, d: 15:41:49, }"] [2025/04/04 15:40:45.545 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 79, b: 79, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:50, }"] [row2="{ a: 79, b: 79, c: hqpB, zFzkK: IsNull, d: 15:41:50, }"] [2025/04/04 15:40:45.646 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 80, b: 80, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:51, }"] [row2="{ a: 80, b: 80, c: 8RgH, zFzkK: IsNull, d: 15:41:51, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:40:45.746 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 81, b: 81, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:52, }"] [row2="{ a: 81, b: 81, c: VVQfaxZ7a, zFzkK: IsNull, d: 15:41:52, }"] [2025/04/04 15:40:45.846 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 82, b: 82, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:53, }"] [row2="{ a: 82, b: 82, c: 7Gql28CzuE, zFzkK: IsNull, d: 15:41:53, }"] [2025/04/04 15:40:45.946 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 83, b: 83, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:54, }"] [row2="{ b: 83, c: mC0X, zFzkK: IsNull, d: 15:41:54, a: 83, }"] [2025/04/04 15:40:46.046 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:55, a: 84, b: 84, c: Nw9p, }"] [row2="{ b: 84, c: LFnvTQ, zFzkK: IsNull, d: 15:41:55, a: 84, }"] [2025/04/04 15:40:46.147 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 85, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:56, a: 85, }"] [row2="{ c: ph, zFzkK: IsNull, d: 15:41:56, a: 85, b: 85, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:45 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041865a0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97874813-4250-4e60-8a9e-3d884296cb87 {"id":"97874813-4250-4e60-8a9e-3d884296cb87","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752442} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd909fd 97874813-4250-4e60-8a9e-3d884296cb87 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/97874813-4250-4e60-8a9e-3d884296cb87/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041865a0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97874813-4250-4e60-8a9e-3d884296cb87 {"id":"97874813-4250-4e60-8a9e-3d884296cb87","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752442} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd909fd 97874813-4250-4e60-8a9e-3d884296cb87 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/97874813-4250-4e60-8a9e-3d884296cb87/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041865a0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231787487246 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97874813-4250-4e60-8a9e-3d884296cb87 {"id":"97874813-4250-4e60-8a9e-3d884296cb87","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752442} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd909fd 97874813-4250-4e60-8a9e-3d884296cb87 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231787487246,"admin-job-type":0} /tidb/cdc/default/default/task/position/97874813-4250-4e60-8a9e-3d884296cb87/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully wait process cdc.test exit for 2-th time... table batch_add_table.finish_mark exists table test.t_1 does not exists check diff successfully nonempty select id, val from test.availability1 where id=2 and val=2 run task successfully wait process cdc.test exit for 1-th time... [2025/04/04 15:40:46.247 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 86, b: 86, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:57, }"] [row2="{ c: j, zFzkK: IsNull, d: 15:41:57, a: 86, b: 86, }"] [2025/04/04 15:40:46.347 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 88, b: 88, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:41:59, }"] [row2="{ a: 88, b: 88, c: PU, zFzkK: IsNull, d: 15:41:59, }"] table test.t2 exists check diff successfully [2025/04/04 15:40:46.447 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:00, a: 89, b: 89, }"] [row2="{ zFzkK: IsNull, d: 15:42:00, a: 89, b: 89, c: 4Zew32vG, }"] [2025/04/04 15:40:46.547 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:42:01, a: 90, b: 90, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:42:01, a: 90, b: 90, c: sKvI00RIL, }"] [2025/04/04 15:40:46.648 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 91, b: 91, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:02, }"] [row2="{ d: 15:42:02, a: 91, b: 91, c: tis5mm, zFzkK: IsNull, }"] start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:46 CST 2025] <<<<<< run test case ddl_reentrant success! >>>>>> wait process cdc.test exit for 1-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_api_tls_with_user_auth/run.sh using Sink-Type: mysql... <<================= table consistent_replicate_storage_file.check1 not exists for 12-th check, retry later wait process cdc.test exit for 2-th time... [2025/04/04 15:40:46.748 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 92, b: 92, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:03, }"] [row2="{ a: 92, b: 92, c: x5hzyaa, zFzkK: IsNull, d: 15:42:03, }"] [2025/04/04 15:40:46.848 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 94, b: 94, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:05, }"] [row2="{ zFzkK: IsNull, d: 15:42:05, a: 94, b: 94, c: y62jbnV, }"] WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Requirement already up-to-date: requests==2.26.0 in /usr/local/lib/python3.6/site-packages Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) [2025/04/04 15:40:46.948 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:42:06, a: 95, b: 95, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 95, b: 95, c: R0eOCc, zFzkK: IsNull, d: 15:42:06, }"] [2025/04/04 15:40:47.048 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:08, a: 97, b: 97, c: Nw9p, }"] [row2="{ d: 15:42:08, a: 97, b: 97, c: rZchW8, zFzkK: IsNull, }"] [2025/04/04 15:40:47.149 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 98, b: 98, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:09, }"] [row2="{ d: 15:42:09, a: 98, b: 98, c: F, zFzkK: IsNull, }"] wait process cdc.test exit for 1-th time... Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_hang_up_capture [Fri Apr 4 15:40:47 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.59075909.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:40:47.249 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 99, b: 99, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:10, }"] [row2="{ zFzkK: IsNull, d: 15:42:10, a: 99, b: 99, c: eLlgnlvfH0, }"] [2025/04/04 15:40:47.349 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 100, b: 100, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:11, }"] [row2="{ zFzkK: IsNull, d: 15:42:11, a: 100, b: 100, c: Ma4yCs4S, }"] Requirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) The 1 times to try to start tidb cluster... [2025/04/04 15:40:47.449 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:12, a: 101, b: 101, }"] [row2="{ zFzkK: IsNull, d: 15:42:12, a: 101, b: 101, c: otjDw6icV, }"] [2025/04/04 15:40:47.549 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 102, b: 102, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:13, }"] [row2="{ a: 102, b: 102, c: l, zFzkK: IsNull, d: 15:42:13, }"] [2025/04/04 15:40:47.650 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 103, b: 103, c: HQ, zFzkK: IsNull, d: 15:42:14, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:40:47.750 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 104, b: 104, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:15, }"] [row2="{ d: 15:42:15, a: 104, b: 104, c: qmvF7vdD, zFzkK: IsNull, }"] [2025/04/04 15:40:47.850 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 105, b: 105, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:39, }"] [row2="{ a: 105, b: 105, c: AgKtb, zFzkK: IsNull, d: 15:45:39, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:47 CST 2025] <<<<<< run test case batch_add_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:47.950 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 106, b: 106, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:28, }"] [row2="{ c: EqBGO06wT, zFzkK: IsNull, d: 15:45:28, a: 106, b: 106, }"] [2025/04/04 15:40:48.051 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 107, b: 107, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:56, }"] [row2="{ a: 107, b: 107, c: ZHea3, zFzkK: IsNull, d: 15:48:56, }"] [2025/04/04 15:40:48.151 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:19, a: 108, b: 108, c: Nw9p, }"] [row2="{ a: 108, b: 108, c: R, zFzkK: IsNull, d: 15:42:19, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:40:48 CST 2025] <<<<<< run test case resolve_lock success! >>>>>> [2025/04/04 15:40:48.251 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 109, b: 109, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:20, }"] [row2="{ b: 109, c: 4FVH, zFzkK: IsNull, d: 15:42:20, a: 109, }"] [2025/04/04 15:40:48.351 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:42:23, a: 112, b: 112, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: B, zFzkK: IsNull, d: 15:42:23, a: 112, b: 112, }"] [2025/04/04 15:40:48.451 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:22, a: 113, b: 113, c: Nw9p, }"] [row2="{ d: 15:47:22, a: 113, b: 113, c: QI5nbj, zFzkK: IsNull, }"] [2025/04/04 15:40:48.551 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 114, b: 114, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:25, }"] [row2="{ c: Y8BjhOvUJC, zFzkK: IsNull, d: 15:42:25, a: 114, b: 114, }"] [2025/04/04 15:40:48.652 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 115, b: 115, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:26, }"] [row2="{ c: A8hli, zFzkK: IsNull, d: 15:42:26, a: 115, b: 115, }"] table consistent_replicate_storage_file.check1 not exists for 13-th check, retry later [2025/04/04 15:40:48.752 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 116, b: 116, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:27, }"] [row2="{ d: 15:42:27, a: 116, b: 116, c: 8TAKYn, zFzkK: IsNull, }"] [2025/04/04 15:40:48.852 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 118, b: 118, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:29, }"] [row2="{ a: 118, b: 118, c: Zl04uUOh, zFzkK: IsNull, d: 15:42:29, }"] start tidb cluster in /tmp/tidb_cdc_test/syncpoint_check_ts Starting Upstream PD... [2025/04/04 15:40:48.952 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 120, b: 120, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:31, }"] [row2="{ a: 120, b: 120, c: n4jgkGQ5xh, zFzkK: IsNull, d: 15:42:31, }"] [2025/04/04 15:40:49.052 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 121, b: 121, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:32, }"] [row2="{ c: 68g, zFzkK: IsNull, d: 15:42:32, a: 121, b: 121, }"] [2025/04/04 15:40:49.153 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 122, b: 122, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:33, }"] [row2="{ b: 122, c: WJE3vcI, zFzkK: IsNull, d: 15:42:33, a: 122, }"] + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:40:49.253 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 123, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:34, a: 123, }"] [row2="{ a: 123, b: 123, c: 5gyCWbS, zFzkK: IsNull, d: 15:42:34, }"] [2025/04/04 15:40:49.353 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 125, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:36, a: 125, }"] [row2="{ a: 125, b: 125, c: dh1W1, zFzkK: IsNull, d: 15:42:36, }"] [2025/04/04 15:40:49.453 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:37, a: 126, b: 126, c: Nw9p, }"] [row2="{ zFzkK: IsNull, d: 15:42:37, a: 126, b: 126, c: WHBjaE, }"] [2025/04/04 15:40:49.554 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 127, b: 127, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:38, }"] [row2="{ a: 127, b: 127, c: H0y1, zFzkK: IsNull, d: 15:42:38, }"] [2025/04/04 15:40:49.655 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:42:39, a: 128, b: 128, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:42:39, a: 128, b: 128, c: RA, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359713796286682 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb1bc40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:9825, start at 2025-04-04 15:40:47.737614534 +0800 CST m=+2.298068675 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:47.745 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:47.729 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:47.729 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359713796286682 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb1bc40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:9825, start at 2025-04-04 15:40:47.737614534 +0800 CST m=+2.298068675 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:47.745 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:47.729 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:47.729 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359713416119596 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb1d54000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:9892, start at 2025-04-04 15:40:47.862482945 +0800 CST m=+2.322129147 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:47.870 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:47.873 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:47.873 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_with_auth/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:40:49.755 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:43, a: 132, b: 132, c: Nw9p, }"] [row2="{ a: 132, b: 132, c: l, zFzkK: IsNull, d: 15:42:43, }"] [2025/04/04 15:40:49.855 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 134, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:45, a: 134, }"] [row2="{ zFzkK: IsNull, d: 15:42:45, a: 134, b: 134, c: GaZCj, }"] [2025/04/04 15:40:49.955 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 135, b: 135, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:46, }"] [row2="{ a: 135, b: 135, c: r, zFzkK: IsNull, d: 15:42:46, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:40:50.055 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 136, b: 136, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:47, }"] [row2="{ b: 136, c: tDrisyyi, zFzkK: IsNull, d: 15:42:47, a: 136, }"] [2025/04/04 15:40:50.155 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 137, b: 137, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:48, }"] [row2="{ zFzkK: IsNull, d: 15:42:48, a: 137, b: 137, c: Fq4u16KoCK, }"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [2025/04/04 15:40:50.255 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:42:49, a: 138, b: 138, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 138, b: 138, c: RjE10Yu8gQ, zFzkK: IsNull, d: 15:42:49, }"] [2025/04/04 15:40:50.356 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 139, b: 139, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:50, }"] [row2="{ b: 139, c: YW, zFzkK: IsNull, d: 15:42:50, a: 139, }"] [2025/04/04 15:40:50.456 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 140, b: 140, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:51, }"] [row2="{ a: 140, b: 140, c: HIhqt6, zFzkK: IsNull, d: 15:42:51, }"] [Pipeline] // dir [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:50 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e79a70 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114240965935107 MinTableBarrierTs:457114242276392965 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c608164a-7682-4b65-88f0-d472ffb8977d {"id":"c608164a-7682-4b65-88f0-d472ffb8977d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe53b c608164a-7682-4b65-88f0-d472ffb8977d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114240965935107,"min-table-barrier-ts":457114242276392965,"admin-job-type":0} /tidb/cdc/default/default/task/position/c608164a-7682-4b65-88f0-d472ffb8977d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e79a70 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114240965935107 MinTableBarrierTs:457114242276392965 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c608164a-7682-4b65-88f0-d472ffb8977d {"id":"c608164a-7682-4b65-88f0-d472ffb8977d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe53b c608164a-7682-4b65-88f0-d472ffb8977d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114240965935107,"min-table-barrier-ts":457114242276392965,"admin-job-type":0} /tidb/cdc/default/default/task/position/c608164a-7682-4b65-88f0-d472ffb8977d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e79a70 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114240965935107 MinTableBarrierTs:457114242276392965 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114240965935107, checkpointTs: 457114240965935107, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c608164a-7682-4b65-88f0-d472ffb8977d {"id":"c608164a-7682-4b65-88f0-d472ffb8977d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe53b c608164a-7682-4b65-88f0-d472ffb8977d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114240965935107,"min-table-barrier-ts":457114242276392965,"admin-job-type":0} /tidb/cdc/default/default/task/position/c608164a-7682-4b65-88f0-d472ffb8977d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Pipeline] // withCredentials [2025/04/04 15:40:50.556 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:52, a: 141, b: 141, c: Nw9p, }"] [row2="{ b: 141, c: tum684bS, zFzkK: IsNull, d: 15:42:52, a: 141, }"] [2025/04/04 15:40:50.656 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:42:53, a: 142, b: 142, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ d: 15:42:53, a: 142, b: 142, c: jMRgRACMm, zFzkK: IsNull, }"] [Pipeline] } [Pipeline] // timeout [Pipeline] } [2025/04/04 15:40:50.756 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 143, b: 143, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:54, }"] [row2="{ a: 143, b: 143, c: blO1, zFzkK: IsNull, d: 15:42:54, }"] [2025/04/04 15:40:50.857 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:55, a: 144, b: 144, c: Nw9p, }"] [row2="{ a: 144, b: 144, c: OpvC7NVU, zFzkK: IsNull, d: 15:42:55, }"] [2025/04/04 15:40:50.957 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 145, b: 145, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:56, }"] [row2="{ b: 145, c: 6iZujmWB, zFzkK: IsNull, d: 15:42:56, a: 145, }"] [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [2025/04/04 15:40:51.057 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:57, a: 146, b: 146, c: Nw9p, }"] [row2="{ c: NiYY, zFzkK: IsNull, d: 15:42:57, a: 146, b: 146, }"] [2025/04/04 15:40:51.157 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 148, b: 148, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:42:59, }"] [row2="{ a: 148, b: 148, c: B5yORIU, zFzkK: IsNull, d: 15:42:59, }"] table consistent_replicate_storage_file.check1 not exists for 14-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [2025/04/04 15:40:51.257 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 149, b: 149, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:00, }"] [row2="{ d: 15:43:00, a: 149, b: 149, c: ZqgeOMP, zFzkK: IsNull, }"] [2025/04/04 15:40:51.357 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ d: 15:43:01, a: 150, b: 150, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 150, b: 150, c: a, zFzkK: IsNull, d: 15:43:01, }"] [2025/04/04 15:40:51.458 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 151, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:56, a: 151, }"] [row2="{ a: 151, b: 151, c: Qq4USDKC, zFzkK: IsNull, d: 15:43:02, }"] [Pipeline] } Verifying downstream PD is started... [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2025/04/04 15:40:51.558 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 152, b: 152, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:03, }"] [row2="{ zFzkK: IsNull, d: 15:43:03, a: 152, b: 152, c: OJnVi9iNQB, }"] [2025/04/04 15:40:51.658 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:43:04, a: 153, b: 153, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: 8Fvejx1JAS, zFzkK: IsNull, d: 15:43:04, a: 153, b: 153, }"] [Pipeline] // stage start tidb cluster in /tmp/tidb_cdc_test/http_api_tls_with_user_auth Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [Pipeline] } Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:51.758 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:43:05, a: 154, b: 154, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: 1C, zFzkK: IsNull, d: 15:43:05, a: 154, b: 154, }"] [2025/04/04 15:40:51.858 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 155, b: 155, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:06, }"] [row2="{ a: 155, b: 155, c: qHf2xKpZ8, zFzkK: IsNull, d: 15:43:06, }"] [2025/04/04 15:40:51.958 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:27, a: 156, b: 156, }"] [row2="{ d: 15:45:27, a: 156, b: 156, c: Bua, zFzkK: IsNull, }"] "is-owner": true, run task successfully [2025/04/04 15:40:52.059 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 157, b: 157, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:24, }"] [row2="{ d: 15:43:08, a: 157, b: 157, c: HUfcXYzjpz, zFzkK: IsNull, }"] [2025/04/04 15:40:52.159 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:09, a: 158, b: 158, }"] [row2="{ a: 158, b: 158, c: CK, zFzkK: IsNull, d: 15:43:09, }"] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11212.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:40:52.259 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:10, a: 159, b: 159, }"] [row2="{ a: 159, b: 159, c: zb4aKdQ, zFzkK: IsNull, d: 15:43:10, }"] [2025/04/04 15:40:52.359 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 161, b: 161, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:12, }"] [row2="{ a: 161, b: 161, c: 3MtQ0Tqp, zFzkK: IsNull, d: 15:43:12, }"] [2025/04/04 15:40:52.459 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 162, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:13, a: 162, }"] [row2="{ zFzkK: IsNull, d: 15:43:13, a: 162, b: 162, c: wJaU7hS5A, }"] chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:40:52.560 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:14, a: 163, b: 163, }"] [row2="{ zFzkK: IsNull, d: 15:43:14, a: 163, b: 163, c: aKo0z, }"] [2025/04/04 15:40:52.660 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 164, b: 164, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:15, }"] [row2="{ d: 15:43:15, a: 164, b: 164, c: B8arwv2ZW, zFzkK: IsNull, }"] [2025/04/04 15:40:52.760 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 165, b: 165, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:16, }"] [row2="{ a: 165, b: 165, c: StgfCk3Fv, zFzkK: IsNull, d: 15:43:16, }"] [2025/04/04 15:40:52.860 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 166, b: 166, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:17, }"] [row2="{ a: 166, b: 166, c: ZJJmeor, zFzkK: IsNull, d: 15:43:17, }"] [2025/04/04 15:40:52.960 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 167, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:08, a: 167, }"] [row2="{ zFzkK: IsNull, d: 15:49:08, a: 167, b: 167, c: ur, }"] table consistent_replicate_storage_file.check1 not exists for 15-th check, retry later [2025/04/04 15:40:53.060 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 168, b: 168, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:19, }"] [row2="{ d: 15:43:19, a: 168, b: 168, c: ew, zFzkK: IsNull, }"] [2025/04/04 15:40:53.161 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 169, b: 169, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:20, }"] [row2="{ a: 169, b: 169, c: 3yKpNbZD, zFzkK: IsNull, d: 15:43:20, }"] owner pid: 5912 owner id c608164a-7682-4b65-88f0-d472ffb8977d [Fri Apr 4 15:40:53 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.60216023.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:40:53.261 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 170, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:21, a: 170, }"] [row2="{ a: 170, b: 170, c: Hkn, zFzkK: IsNull, d: 15:43:21, }"] [2025/04/04 15:40:53.361 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 171, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:22, a: 171, }"] [row2="{ c: wVIJlN7Z6, zFzkK: IsNull, d: 15:43:22, a: 171, b: 171, }"] [2025/04/04 15:40:53.461 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 172, b: 172, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:23, }"] [row2="{ a: 172, b: 172, c: 1uf, zFzkK: IsNull, d: 15:43:23, }"] [2025/04/04 15:40:53.561 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:24, a: 173, b: 173, }"] [row2="{ a: 173, b: 173, c: cqjMsg, zFzkK: IsNull, d: 15:43:24, }"] [2025/04/04 15:40:53.662 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 175, b: 175, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:26, }"] [row2="{ a: 175, b: 175, c: mDJi, zFzkK: IsNull, d: 15:43:26, }"] + set +x + tso='457114242835283969 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114242835283969 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:53.762 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:27, a: 176, b: 176, c: Nw9p, }"] [row2="{ a: 176, b: 176, c: wrpt5oBrnr, zFzkK: IsNull, d: 15:43:27, }"] [2025/04/04 15:40:53.862 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 177, b: 177, c: Oso, zFzkK: IsNull, d: 15:43:28, }"] [2025/04/04 15:40:53.962 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 178, b: 178, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:29, }"] [row2="{ c: svkBM, zFzkK: IsNull, d: 15:43:29, a: 178, b: 178, }"] [2025/04/04 15:40:54.062 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 179, b: 179, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:30, }"] [row2="{ c: L5CuwEe7, zFzkK: IsNull, d: 15:43:30, a: 179, b: 179, }"] [2025/04/04 15:40:54.162 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 180, b: 180, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:31, }"] [row2="{ d: 15:43:31, a: 180, b: 180, c: T9ic, zFzkK: IsNull, }"] [2025/04/04 15:40:54.263 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 181, b: 181, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:32, }"] [row2="{ a: 181, b: 181, c: I, zFzkK: IsNull, d: 15:43:32, }"] wait process 9258 exit for 1-th time... [2025/04/04 15:40:54.363 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 182, b: 182, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:33, }"] [row2="{ a: 182, b: 182, c: GzdS, zFzkK: IsNull, d: 15:43:33, }"] [2025/04/04 15:40:54.463 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:34, a: 183, b: 183, c: Nw9p, }"] [row2="{ b: 183, c: tL2jiyP, zFzkK: IsNull, d: 15:43:34, a: 183, }"] [2025/04/04 15:40:54.563 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 184, b: 184, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:35, }"] [row2="{ b: 184, c: 8, zFzkK: IsNull, d: 15:43:35, a: 184, }"] [2025/04/04 15:40:54.663 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 185, b: 185, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:36, }"] [row2="{ zFzkK: IsNull, d: 15:43:36, a: 185, b: 185, c: MuPr, }"] [2025/04/04 15:40:54.764 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:43:37, a: 186, b: 186, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:43:37, a: 186, b: 186, c: pECm, }"] wait process 9258 exit for 2-th time... Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [2025/04/04 15:40:54.864 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 187, b: 187, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:38, }"] [row2="{ c: 4jRF, zFzkK: IsNull, d: 15:43:38, a: 187, b: 187, }"] [2025/04/04 15:40:54.964 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 188, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:39, a: 188, }"] [row2="{ a: 188, b: 188, c: f2R4gEk, zFzkK: IsNull, d: 15:43:39, }"] table consistent_replicate_storage_file.check1 not exists for 16-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // cache Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:40:54 CST 2025] <<<<<< START cdc server in cli_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.1125711259.out server --log-file /tmp/tidb_cdc_test/cli_with_auth/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_with_auth/cdc_data --cluster-id default --config /tmp/tidb_cdc_test/cli_with_auth/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] } [2025/04/04 15:40:55.064 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 189, b: 189, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:35, }"] [row2="{ a: 189, b: 189, c: Ruo3cF7P4, zFzkK: IsNull, d: 15:48:35, }"] [2025/04/04 15:40:55.164 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ zFzkK: IsNull, d: 15:43:41, a: 190, b: 190, c: AppQBdaC, }"] [2025/04/04 15:40:55.264 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:42, a: 191, b: 191, }"] [row2="{ a: 191, b: 191, c: m46Gyua, zFzkK: IsNull, d: 15:43:42, }"] [Pipeline] // dir [Pipeline] } Starting Upstream TiDB... /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (9258) - No such process wait process 9258 exit for 3-th time... process 9258 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Pipeline] // withCredentials [Pipeline] } [2025/04/04 15:40:55.365 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:43:43, a: 192, b: 192, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 192, b: 192, c: uDVJpG8J, zFzkK: IsNull, d: 15:43:43, }"] [2025/04/04 15:40:55.465 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 193, b: 193, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:44, }"] [row2="{ a: 193, b: 193, c: 42Hr3GNE, zFzkK: IsNull, d: 15:43:44, }"] [Pipeline] // timeout [Pipeline] } Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:40:55 CST 2025] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.94039405.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // stage [Pipeline] } [2025/04/04 15:40:55.565 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 194, b: 194, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:45, }"] [row2="{ a: 194, b: 194, c: rorKZXX, zFzkK: IsNull, d: 15:43:45, }"] [2025/04/04 15:40:55.665 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 195, b: 195, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:46, }"] [row2="{ a: 195, b: 195, c: ORVpIW8F, zFzkK: IsNull, d: 15:43:46, }"] [2025/04/04 15:40:55.765 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 196, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:47, a: 196, }"] [row2="{ b: 196, c: lJCLrzevh2, zFzkK: IsNull, d: 15:43:47, a: 196, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2025/04/04 15:40:55.865 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 197, b: 197, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:48, }"] [row2="{ b: 197, c: S, zFzkK: IsNull, d: 15:43:48, a: 197, }"] [2025/04/04 15:40:55.966 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 198, b: 198, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:49, }"] [row2="{ zFzkK: IsNull, d: 15:43:49, a: 198, b: 198, c: Rr, }"] [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [2025/04/04 15:40:56.066 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 199, b: 199, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:50, }"] [row2="{ c: yS9X, zFzkK: IsNull, d: 15:43:50, a: 199, b: 199, }"] [2025/04/04 15:40:56.166 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 200, b: 200, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:57, }"] [row2="{ a: 200, b: 200, c: D13o9X, zFzkK: IsNull, d: 15:43:57, }"] [2025/04/04 15:40:56.266 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 201, b: 201, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:58, }"] [row2="{ b: 201, c: Pi9Tmpbc, zFzkK: IsNull, d: 15:43:58, a: 201, }"] [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359737423950279 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb7b7c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:26081, start at 2025-04-04 15:40:53.88149144 +0800 CST m=+2.248746261 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:53.937 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:53.930 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:53.930 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359737423950279 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb7b7c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:26081, start at 2025-04-04 15:40:53.88149144 +0800 CST m=+2.248746261 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:53.937 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:53.930 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:53.930 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359732450331453 TiDB Cluster ID. tikv_gc_leader_uuid 657fefb7d540003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-qfqk9-cn0b1-8c95p, pid:26151, start at 2025-04-04 15:40:53.977023997 +0800 CST m=+2.237489614 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:53.985 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:54.030 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:54.030 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // stage [Pipeline] } [2025/04/04 15:40:56.366 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 202, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:43:59, a: 202, }"] [row2="{ d: 15:43:59, a: 202, b: 202, c: yO9tgQ, zFzkK: IsNull, }"] [2025/04/04 15:40:56.466 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ zFzkK: IsNull, d: 15:44:00, a: 203, b: 203, c: 3WfZscbXOR, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:56 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ba3950 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114243587375113 MinTableBarrierTs:457114243849519113 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b142fcd5-4832-409c-a583-28a28e1f25a8 {"id":"b142fcd5-4832-409c-a583-28a28e1f25a8","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752453} /tidb/cdc/default/__cdc_meta__/capture/c608164a-7682-4b65-88f0-d472ffb8977d {"id":"c608164a-7682-4b65-88f0-d472ffb8977d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe53b c608164a-7682-4b65-88f0-d472ffb8977d /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe588 b142fcd5-4832-409c-a583-28a28e1f25a8 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114243587375113,"min-table-barrier-ts":457114243849519113,"admin-job-type":0} /tidb/cdc/default/default/task/position/b142fcd5-4832-409c-a583-28a28e1f25a8/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/c608164a-7682-4b65-88f0-d472ffb8977d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ba3950 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114243587375113 MinTableBarrierTs:457114243849519113 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b142fcd5-4832-409c-a583-28a28e1f25a8 {"id":"b142fcd5-4832-409c-a583-28a28e1f25a8","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752453} /tidb/cdc/default/__cdc_meta__/capture/c608164a-7682-4b65-88f0-d472ffb8977d {"id":"c608164a-7682-4b65-88f0-d472ffb8977d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe53b c608164a-7682-4b65-88f0-d472ffb8977d /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe588 b142fcd5-4832-409c-a583-28a28e1f25a8 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114243587375113,"min-table-barrier-ts":457114243849519113,"admin-job-type":0} /tidb/cdc/default/default/task/position/b142fcd5-4832-409c-a583-28a28e1f25a8/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/c608164a-7682-4b65-88f0-d472ffb8977d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ba3950 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114243587375113 MinTableBarrierTs:457114243849519113 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b142fcd5-4832-409c-a583-28a28e1f25a8 {"id":"b142fcd5-4832-409c-a583-28a28e1f25a8","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752453} /tidb/cdc/default/__cdc_meta__/capture/c608164a-7682-4b65-88f0-d472ffb8977d {"id":"c608164a-7682-4b65-88f0-d472ffb8977d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752447} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe53b c608164a-7682-4b65-88f0-d472ffb8977d /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe588 b142fcd5-4832-409c-a583-28a28e1f25a8 /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114243587375113,"min-table-barrier-ts":457114243849519113,"admin-job-type":0} /tidb/cdc/default/default/task/position/b142fcd5-4832-409c-a583-28a28e1f25a8/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/c608164a-7682-4b65-88f0-d472ffb8977d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "c608164a-7682-4b65-88f0-d472ffb8977d" | grep id =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_cdc_cluster/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:40:56.567 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 204, b: 204, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:01, }"] [row2="{ d: 15:44:01, a: 204, b: 204, c: AyvBYTADuN, zFzkK: IsNull, }"] [2025/04/04 15:40:56.667 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:02, a: 205, b: 205, }"] [row2="{ d: 15:44:02, a: 205, b: 205, c: pUA2o0P7A, zFzkK: IsNull, }"] [2025/04/04 15:40:56.767 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:03, a: 206, b: 206, }"] [row2="{ d: 15:44:03, a: 206, b: 206, c: DhbNUmj, zFzkK: IsNull, }"] table consistent_replicate_storage_file.check1 not exists for 17-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:56.867 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 207, b: 207, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:04, }"] [row2="{ zFzkK: IsNull, d: 15:44:04, a: 207, b: 207, c: xqwQ5gkgME, }"] [2025/04/04 15:40:56.967 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 208, b: 208, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:05, }"] [row2="{ c: B4qyE0KrJ, zFzkK: IsNull, d: 15:44:05, a: 208, b: 208, }"] [2025/04/04 15:40:57.068 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:06, a: 209, b: 209, }"] [row2="{ a: 209, b: 209, c: Zei, zFzkK: IsNull, d: 15:44:06, }"] [2025/04/04 15:40:57.168 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:07, a: 210, b: 210, }"] [row2="{ a: 210, b: 210, c: M, zFzkK: IsNull, d: 15:44:07, }"] [2025/04/04 15:40:57.268 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 211, b: 211, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:08, }"] [row2="{ a: 211, b: 211, c: XU5jHtZpw5, zFzkK: IsNull, d: 15:44:08, }"] [2025/04/04 15:40:57.368 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ zFzkK: IsNull, d: 15:44:09, a: 212, b: 212, c: NE7YCWUPC, }"] [2025/04/04 15:40:57.468 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 213, b: 213, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:10, }"] [row2="{ b: 213, c: TYRmmgdAi, zFzkK: IsNull, d: 15:44:10, a: 213, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:57.569 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:11, a: 214, b: 214, c: Nw9p, }"] [row2="{ c: FLsTlR, zFzkK: IsNull, d: 15:44:11, a: 214, b: 214, }"] [2025/04/04 15:40:57.669 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 215, b: 215, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:12, }"] [row2="{ d: 15:44:12, a: 215, b: 215, c: i6, zFzkK: IsNull, }"] [2025/04/04 15:40:57.769 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 216, b: 216, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:13, }"] [row2="{ a: 216, b: 216, c: k8RxwJKle4, zFzkK: IsNull, d: 15:44:13, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... "cluster-id": "default" "id": "b142fcd5-4832-409c-a583-28a28e1f25a8", "cluster-id": "default" run task successfully [2025/04/04 15:40:57.869 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ c: 1crJ7pJE, zFzkK: IsNull, d: 15:44:14, a: 217, b: 217, }"] [2025/04/04 15:40:57.969 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:16, a: 219, b: 219, c: Nw9p, }"] [row2="{ zFzkK: IsNull, d: 15:44:16, a: 219, b: 219, c: K1, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359467753540872 TiDB Cluster ID. tikv_gc_leader_uuid 657fef79994000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9146, start at 2025-04-04 15:39:50.273178944 +0800 CST m=+2.246077329 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.281 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed The 3 times to try to start tidb cluster... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.27462.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:40:58.070 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 220, b: 220, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:17, }"] [row2="{ a: 220, b: 220, c: 8eEoKH9Oc9, zFzkK: IsNull, d: 15:44:17, }"] [2025/04/04 15:40:58.170 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 221, b: 221, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:18, }"] [row2="{ a: 221, b: 221, c: STN, zFzkK: IsNull, d: 15:44:18, }"] [2025/04/04 15:40:58.270 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:19, a: 222, b: 222, }"] [row2="{ a: 222, b: 222, c: YCzKdp, zFzkK: IsNull, d: 15:44:19, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:57 GMT < Content-Length: 854 < Content-Type: text/plain; charset=utf-8 < { [854 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad6fbf8f-d684-499e-9384-780b406677df {"id":"ad6fbf8f-d684-499e-9384-780b406677df","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbea9c665 ad6fbf8f-d684-499e-9384-780b406677df /tidb/cdc/default/default/upstream/7489359713796286682 {"id":7489359713796286682,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad6fbf8f-d684-499e-9384-780b406677df {"id":"ad6fbf8f-d684-499e-9384-780b406677df","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbea9c665 ad6fbf8f-d684-499e-9384-780b406677df /tidb/cdc/default/default/upstream/7489359713796286682 {"id":7489359713796286682,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ad6fbf8f-d684-499e-9384-780b406677df {"id":"ad6fbf8f-d684-499e-9384-780b406677df","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbea9c665 ad6fbf8f-d684-499e-9384-780b406677df /tidb/cdc/default/default/upstream/7489359713796286682 {"id":7489359713796286682,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11313.out cli changefeed create --start-ts=457114242835283969 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489359713796286682,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:40:58.228166085+08:00","start_ts":457114242835283969,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114242835283969,"checkpoint_ts":457114242835283969,"checkpoint_time":"2025-04-04 15:40:52.222"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:40:58.370 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 223, b: 223, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:20, }"] [row2="{ c: JPP, zFzkK: IsNull, d: 15:44:20, a: 223, b: 223, }"] [2025/04/04 15:40:58.470 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 224, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:21, a: 224, }"] [row2="{ a: 224, b: 224, c: o, zFzkK: IsNull, d: 15:44:21, }"] [2025/04/04 15:40:58.571 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 225, b: 225, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:22, }"] [row2="{ a: 225, b: 225, c: mpIFW1j, zFzkK: IsNull, d: 15:44:22, }"] [2025/04/04 15:40:58.671 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:44:23, a: 226, b: 226, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 226, b: 226, c: d, zFzkK: IsNull, d: 15:44:23, }"] [2025/04/04 15:40:58.771 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 227, b: 227, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:24, }"] [row2="{ a: 227, b: 227, c: Nv, zFzkK: IsNull, d: 15:44:24, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:40:58 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001271710 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231931666433 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1 {"id":"422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd90a9b 422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231931666433,"admin-job-type":0} /tidb/cdc/default/default/task/position/422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001271710 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231931666433 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1 {"id":"422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd90a9b 422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231931666433,"admin-job-type":0} /tidb/cdc/default/default/task/position/422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {UpstreamID:7489359484104304026 Namespace:default ID:5957e1d6-26d3-4fd6-b99e-b5430e140c29 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:40:02.094338046 +0800 CST StartTs:457114229677228034 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc001271710 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229690335235} {CheckpointTs:457114231787487246 MinTableBarrierTs:457114231931666433 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114231787487246, checkpointTs: 457114231787487246, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1 {"id":"422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752455} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdd90a9b 422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1 /tidb/cdc/default/default/changefeed/info/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"upstream-id":7489359484104304026,"namespace":"default","changefeed-id":"5957e1d6-26d3-4fd6-b99e-b5430e140c29","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:40:02.094338046+08:00","start-ts":457114229677228034,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229690335235} /tidb/cdc/default/default/changefeed/status/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":457114231787487246,"min-table-barrier-ts":457114231931666433,"admin-job-type":0} /tidb/cdc/default/default/task/position/422e749d-f6e4-4bf0-9dfd-9a8dfcbcbfa1/5957e1d6-26d3-4fd6-b99e-b5430e140c29 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359484104304026 {"id":7489359484104304026,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table kill_owner_with_ddl.t1 exists table kill_owner_with_ddl.t2 exists table kill_owner_with_ddl.t3 not exists for 1-th check, retry later [2025/04/04 15:40:58.871 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 228, b: 228, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:25, }"] [row2="{ a: 228, b: 228, c: RTt, zFzkK: IsNull, d: 15:44:25, }"] [2025/04/04 15:40:58.971 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 229, b: 229, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:26, }"] [row2="{ b: 229, c: c9So1zaN, zFzkK: IsNull, d: 15:44:26, a: 229, }"] [2025/04/04 15:40:59.072 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 231, b: 231, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:07, }"] [row2="{ b: 231, c: eOF, zFzkK: IsNull, d: 15:49:07, a: 231, }"] [2025/04/04 15:40:59.172 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 232, b: 232, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:29, }"] [row2="{ b: 232, c: zMPQsb89Q, zFzkK: IsNull, d: 15:44:29, a: 232, }"] [2025/04/04 15:40:59.272 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ b: 233, c: Bgm, zFzkK: IsNull, d: 15:44:30, a: 233, }"] nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table consistent_replicate_storage_file.check1 not exists for 18-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359750431305597 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbab48000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:12707, start at 2025-04-04 15:40:56.946553742 +0800 CST m=+2.160106417 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:56.954 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:56.914 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:56.914 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359750431305597 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbab48000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:12707, start at 2025-04-04 15:40:56.946553742 +0800 CST m=+2.160106417 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:56.954 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:56.914 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:56.914 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359748397273769 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbac20000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:12783, start at 2025-04-04 15:40:56.992426791 +0800 CST m=+2.124908863 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:57.032 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:57.018 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:57.018 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/syncpoint_check_ts/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/syncpoint_check_ts/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:40:59.372 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:31, a: 234, b: 234, }"] [row2="{ a: 234, b: 234, c: MV6rcLW, zFzkK: IsNull, d: 15:44:31, }"] [2025/04/04 15:40:59.472 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 235, b: 235, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:32, }"] [row2="{ c: nLq9aKQbe1, zFzkK: IsNull, d: 15:44:32, a: 235, b: 235, }"] [2025/04/04 15:40:59.573 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:44:33, a: 236, b: 236, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 236, b: 236, c: Q5RmFrM, zFzkK: IsNull, d: 15:44:33, }"] + set +x table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later [2025/04/04 15:40:59.673 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 237, b: 237, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:34, }"] [row2="{ a: 237, b: 237, c: xmDNF, zFzkK: IsNull, d: 15:44:34, }"] [2025/04/04 15:40:59.773 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 238, b: 238, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:35, }"] [row2="{ zFzkK: IsNull, d: 15:44:35, a: 238, b: 238, c: 9c, }"] + set +x + tso='457114244469227521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114244469227521 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:40:59 CST 2025] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.2750327505.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:40:59.873 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:36, a: 239, b: 239, }"] [row2="{ a: 239, b: 239, c: Q6eDix, zFzkK: IsNull, d: 15:44:36, }"] [2025/04/04 15:40:59.974 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ d: 15:44:37, a: 240, b: 240, c: 0xm0x9tC, zFzkK: IsNull, }"] [2025/04/04 15:41:00.074 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 241, b: 241, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:38, }"] [row2="{ d: 15:44:38, a: 241, b: 241, c: LEmSto, zFzkK: IsNull, }"] [2025/04/04 15:41:00.174 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:39, a: 242, b: 242, }"] [row2="{ b: 242, c: nBZkrxm, zFzkK: IsNull, d: 15:44:39, a: 242, }"] [2025/04/04 15:41:00.274 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 243, b: 243, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:40, }"] [row2="{ a: 243, b: 243, c: rn33g4Ta9H, zFzkK: IsNull, d: 15:44:40, }"] [2025/04/04 15:41:00.374 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 244, b: 244, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:41, }"] [row2="{ b: 244, c: o8jTC1ACL, zFzkK: IsNull, d: 15:44:41, a: 244, }"] [2025/04/04 15:41:00.474 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 245, b: 245, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:42, }"] [row2="{ a: 245, b: 245, c: etLfMpQfqv, zFzkK: IsNull, d: 15:44:42, }"] [2025/04/04 15:41:00.575 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 246, b: 246, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:43, }"] [row2="{ zFzkK: IsNull, d: 15:44:43, a: 246, b: 246, c: oACzS, }"] table kill_owner_with_ddl.t3 exists [2025/04/04 15:41:00.675 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 247, b: 247, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:44, }"] [row2="{ b: 247, c: xjkfYY, zFzkK: IsNull, d: 15:44:44, a: 247, }"] [2025/04/04 15:41:00.775 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 248, b: 248, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:45, }"] [row2="{ d: 15:44:45, a: 248, b: 248, c: V, zFzkK: IsNull, }"] start tidb cluster in /tmp/tidb_cdc_test/multi_cdc_cluster Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... check diff failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/multi_rocks Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:41:00.875 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 249, b: 249, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:46, }"] [row2="{ zFzkK: IsNull, d: 15:44:46, a: 249, b: 249, c: Sx, }"] [2025/04/04 15:41:00.975 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:47, a: 250, b: 250, }"] [row2="{ a: 250, b: 250, c: 23Ke2rb, zFzkK: IsNull, d: 15:44:47, }"] [2025/04/04 15:41:01.076 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:44:48, a: 251, b: 251, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:44:48, a: 251, b: 251, c: 3soQY4v, }"] table consistent_replicate_storage_file.check1 exists table consistent_replicate_storage_file.t2 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359761219851327 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbcd50000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:36235, start at 2025-04-04 15:40:59.107449996 +0800 CST m=+2.431039967 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:59.160 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:59.157 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:59.157 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359761219851327 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbcd50000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:36235, start at 2025-04-04 15:40:59.107449996 +0800 CST m=+2.431039967 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:59.160 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:59.157 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:59.157 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359750367368663 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbced80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:36316, start at 2025-04-04 15:40:59.202116667 +0800 CST m=+2.424989947 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:59.210 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:59.190 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:59.190 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_api_tls_with_user_auth/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint_check_ts.cli.14037.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:41:01.176 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 252, b: 252, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:49, }"] [row2="{ a: 252, b: 252, c: F, zFzkK: IsNull, d: 15:44:49, }"] [2025/04/04 15:41:01.276 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:50, a: 253, b: 253, }"] [row2="{ a: 253, b: 253, c: O4ygyxpC, zFzkK: IsNull, d: 15:44:50, }"] start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:41:01.376 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:51, a: 254, b: 254, c: Nw9p, }"] [row2="{ zFzkK: IsNull, d: 15:44:51, a: 254, b: 254, c: bdfuj, }"] [2025/04/04 15:41:01.476 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 255, b: 255, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:52, }"] [row2="{ a: 255, b: 255, c: KaiuWmr, zFzkK: IsNull, d: 15:44:52, }"] [2025/04/04 15:41:01.577 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ d: 15:44:53, a: 256, b: 256, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 256, b: 256, c: l, zFzkK: IsNull, d: 15:44:53, }"] nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later [2025/04/04 15:41:01.677 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 257, b: 257, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:54, }"] [row2="{ a: 257, b: 257, c: XTPGP0mtV, zFzkK: IsNull, d: 15:44:54, }"] [2025/04/04 15:41:01.777 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:55, a: 258, b: 258, }"] [row2="{ a: 258, b: 258, c: Ltreh4u, zFzkK: IsNull, d: 15:44:55, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359764150449268 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbd72c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:12529, start at 2025-04-04 15:40:59.728555299 +0800 CST m=+2.276230982 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:59.736 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:59.723 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:59.723 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359764150449268 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbd72c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:12529, start at 2025-04-04 15:40:59.728555299 +0800 CST m=+2.276230982 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:59.736 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:59.723 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:59.723 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359755295577208 TiDB Cluster ID. tikv_gc_leader_uuid 657fefbd7140004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:12605, start at 2025-04-04 15:40:59.730188438 +0800 CST m=+2.184628481 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:42:59.739 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:40:59.717 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:30:59.717 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:41:01.877 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 259, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:56, a: 259, }"] [row2="{ a: 259, b: 259, c: Kg88k0, zFzkK: IsNull, d: 15:44:56, }"] [2025/04/04 15:41:01.977 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 260, b: 260, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:57, }"] [row2="{ b: 260, c: mf4, zFzkK: IsNull, d: 15:44:57, a: 260, }"] [2025/04/04 15:41:02.077 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:58, a: 261, b: 261, c: Nw9p, }"] [row2="{ zFzkK: IsNull, d: 15:44:58, a: 261, b: 261, c: FkIoL, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table test.`simple-dash` not exists for 2-th check, retry later [2025/04/04 15:41:02.178 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:44:59, a: 262, b: 262, c: Nw9p, }"] [row2="{ d: 15:44:59, a: 262, b: 262, c: toKJj, zFzkK: IsNull, }"] [2025/04/04 15:41:02.278 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 263, b: 263, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:00, }"] [row2="{ a: 263, b: 263, c: GEizRo, zFzkK: IsNull, d: 15:45:00, }"] [2025/04/04 15:41:02.378 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:01, a: 264, b: 264, }"] [row2="{ a: 264, b: 264, c: cO9bKvSl3, zFzkK: IsNull, d: 15:45:01, }"] [2025/04/04 15:41:02.478 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 265, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:02, a: 265, }"] [row2="{ a: 265, b: 265, c: yMhGInx2f, zFzkK: IsNull, d: 15:45:02, }"] [2025/04/04 15:41:02.578 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 266, b: 266, c: w78Yul, zFzkK: IsNull, d: 15:45:03, }"] [2025/04/04 15:41:02.679 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ zFzkK: IsNull, d: 15:45:04, a: 267, b: 267, c: z, }"] [2025/04/04 15:41:02.779 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:05, a: 268, b: 268, }"] [row2="{ b: 268, c: 75c, zFzkK: IsNull, d: 15:45:05, a: 268, }"] [2025/04/04 15:41:02.879 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ b: 269, c: eoyEh, zFzkK: IsNull, d: 15:45:06, a: 269, }"] [2025/04/04 15:41:02.979 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 270, b: 270, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:07, }"] [row2="{ a: 270, b: 270, c: CR1Hl0, zFzkK: IsNull, d: 15:45:07, }"] [2025/04/04 15:41:03.079 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 271, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:08, a: 271, }"] [row2="{ b: 271, c: z9oGR2Hg, zFzkK: IsNull, d: 15:45:08, a: 271, }"] table consistent_replicate_storage_file.t2 not exists for 2-th check, retry later + set +x + tso='457114245284233218 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114245284233218 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:03 CST 2025] <<<<<< START cdc server in syncpoint_check_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint_check_ts.1406714069.out server --log-file /tmp/tidb_cdc_test/syncpoint_check_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/syncpoint_check_ts/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls_with_user_auth Starting TLS PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:02 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cde82009-64de-494e-a719-30760f1a845d {"id":"cde82009-64de-494e-a719-30760f1a845d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbec221e5 cde82009-64de-494e-a719-30760f1a845d /tidb/cdc/default/default/upstream/7489359737423950279 {"id":7489359737423950279,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cde82009-64de-494e-a719-30760f1a845d {"id":"cde82009-64de-494e-a719-30760f1a845d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbec221e5 cde82009-64de-494e-a719-30760f1a845d /tidb/cdc/default/default/upstream/7489359737423950279 {"id":7489359737423950279,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cde82009-64de-494e-a719-30760f1a845d {"id":"cde82009-64de-494e-a719-30760f1a845d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752460} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbec221e5 cde82009-64de-494e-a719-30760f1a845d /tidb/cdc/default/default/upstream/7489359737423950279 {"id":7489359737423950279,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.27562.out cli changefeed create --start-ts=457114244469227521 --sink-uri=mysql://root@127.0.0.1:3306/ [2025/04/04 15:41:03.179 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:45:09, a: 272, b: 272, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 272, b: 272, c: 5xKY, zFzkK: IsNull, d: 15:45:09, }"] [2025/04/04 15:41:03.280 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 273, b: 273, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:10, }"] [row2="{ a: 273, b: 273, c: POPxbKO, zFzkK: IsNull, d: 15:45:10, }"] [2025/04/04 15:41:03.380 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 274, b: 274, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:11, }"] [row2="{ zFzkK: IsNull, d: 15:45:11, a: 274, b: 274, c: V, }"] check diff successfully wait process cdc.test exit for 1-th time... Create changefeed successfully! ID: 1a343b9a-167d-47be-a752-5e57bdd29e80 Info: {"upstream_id":7489359737423950279,"namespace":"default","id":"1a343b9a-167d-47be-a752-5e57bdd29e80","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2025-04-04T15:41:03.401802807+08:00","start_ts":457114244469227521,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114244469227521,"checkpoint_ts":457114244469227521,"checkpoint_time":"2025-04-04 15:40:58.455"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:03.480 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:12, a: 275, b: 275, }"] [row2="{ a: 275, b: 275, c: VVccDRHo, zFzkK: IsNull, d: 15:45:12, }"] [2025/04/04 15:41:03.580 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:48:03, a: 276, b: 276, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 276, b: 276, c: DJTGpO, zFzkK: IsNull, d: 15:48:03, }"] [2025/04/04 15:41:03.680 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 278, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:15, a: 278, }"] [row2="{ zFzkK: IsNull, d: 15:45:15, a: 278, b: 278, c: wxTxYOXo, }"] [2025/04/04 15:41:03.781 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 279, b: 279, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:16, }"] [row2="{ b: 279, c: ghSrs, zFzkK: IsNull, d: 15:45:16, a: 279, }"] [2025/04/04 15:41:03.881 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:17, a: 280, b: 280, c: Nw9p, }"] [row2="{ a: 280, b: 280, c: 0PI, zFzkK: IsNull, d: 15:45:17, }"] + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.13977.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114245784666114, "checkpoint_time": "2025-04-04 15:41:03.473", "error": null }' + echo '{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114245784666114, "checkpoint_time": "2025-04-04 15:41:03.473", "error": null }' { "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114245784666114, "checkpoint_time": "2025-04-04 15:41:03.473", "error": null } ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114245784666114, '"checkpoint_time":' '"2025-04-04' '15:41:03.473",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114245784666114, '"checkpoint_time":' '"2025-04-04' '15:41:03.473",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 [2025/04/04 15:41:03.981 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:18, a: 281, b: 281, c: Nw9p, }"] [row2="{ c: s83D9G6, zFzkK: IsNull, d: 15:45:18, a: 281, b: 281, }"] [2025/04/04 15:41:04.081 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 282, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:19, a: 282, }"] [row2="{ d: 15:45:19, a: 282, b: 282, c: lTJ0le, zFzkK: IsNull, }"] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 [2025/04/04 15:41:04.181 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:20, a: 283, b: 283, c: Nw9p, }"] [row2="{ c: 7iZBwjY, zFzkK: IsNull, d: 15:45:20, a: 283, b: 283, }"] [2025/04/04 15:41:04.282 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 284, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:21, a: 284, }"] [row2="{ zFzkK: IsNull, d: 15:45:21, a: 284, b: 284, c: Wm9hb, }"] [2025/04/04 15:41:04.382 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 285, b: 285, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:22, }"] [row2="{ b: 285, c: MtqKUyjv, zFzkK: IsNull, d: 15:45:22, a: 285, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:04 CST 2025] <<<<<< run test case kill_owner_with_ddl success! >>>>>> changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 [2025/04/04 15:41:04.482 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 286, b: 286, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:23, }"] [row2="{ a: 286, b: 286, c: 8s5BkFYKVf, zFzkK: IsNull, d: 15:45:23, }"] [2025/04/04 15:41:04.582 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 287, b: 287, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:24, }"] [row2="{ d: 15:45:24, a: 287, b: 287, c: LPKLoMxn7, zFzkK: IsNull, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:04.682 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 288, b: 288, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:25, }"] [row2="{ a: 288, b: 288, c: sC2, zFzkK: IsNull, d: 15:45:25, }"] [2025/04/04 15:41:04.783 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 289, b: 289, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:26, }"] [row2="{ c: f, zFzkK: IsNull, d: 15:45:26, a: 289, b: 289, }"] [2025/04/04 15:41:04.883 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 290, b: 290, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:29, }"] [row2="{ zFzkK: IsNull, d: 15:45:29, a: 290, b: 290, c: g55b07abWL, }"] + set +x table consistent_replicate_storage_file.t2 not exists for 3-th check, retry later [2025/04/04 15:41:04.983 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 291, b: 291, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:30, }"] [row2="{ b: 291, c: 975o4Nm98, zFzkK: IsNull, d: 15:45:30, a: 291, }"] [2025/04/04 15:41:05.083 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:31, a: 292, b: 292, c: Nw9p, }"] [row2="{ a: 292, b: 292, c: uP1j, zFzkK: IsNull, d: 15:45:31, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used [2025/04/04 15:41:05.183 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 293, b: 293, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:32, }"] [row2="{ b: 293, c: D, zFzkK: IsNull, d: 15:45:32, a: 293, }"] [2025/04/04 15:41:05.283 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 294, b: 294, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:33, }"] [row2="{ a: 294, b: 294, c: h4Ci2, zFzkK: IsNull, d: 15:45:33, }"] [2025/04/04 15:41:05.384 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 295, b: 295, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:34, }"] [row2="{ zFzkK: IsNull, d: 15:45:34, a: 295, b: 295, c: MarYpDag, }"] Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11598.out cli changefeed --changefeed-id custom-changefeed-name pause Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:41:05.484 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 296, b: 296, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:35, }"] [row2="{ a: 296, b: 296, c: pc2QeLB, zFzkK: IsNull, d: 15:45:35, }"] [2025/04/04 15:41:05.584 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 297, b: 297, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:36, }"] [row2="{ a: 297, b: 297, c: G0qMM, zFzkK: IsNull, d: 15:45:36, }"] PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + set +x + tso='457114245981536257 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114245981536257 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114245981536257 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:41:05 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1401214014.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:05.684 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 298, b: 298, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:37, }"] [row2="{ d: 15:45:37, a: 298, b: 298, c: UicJfWS, zFzkK: IsNull, }"] [2025/04/04 15:41:05.784 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ b: 299, c: d1, zFzkK: IsNull, d: 15:45:38, a: 299, }"] [2025/04/04 15:41:05.885 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 301, b: 301, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:41, }"] [row2="{ b: 301, c: qEIM, zFzkK: IsNull, d: 15:45:41, a: 301, }"] nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later [2025/04/04 15:41:05.985 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 302, b: 302, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:42, }"] [row2="{ a: 302, b: 302, c: w, zFzkK: IsNull, d: 15:45:42, }"] [2025/04/04 15:41:06.085 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 303, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:43, a: 303, }"] [row2="{ zFzkK: IsNull, d: 15:45:43, a: 303, b: 303, c: dyV4, }"] [2025/04/04 15:41:06.185 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 304, b: 304, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:44, }"] [row2="{ a: 304, b: 304, c: X1, zFzkK: IsNull, d: 15:45:44, }"] [2025/04/04 15:41:06.285 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 305, b: 305, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:45, }"] [row2="{ a: 305, b: 305, c: olGcQnw8b4, zFzkK: IsNull, d: 15:45:45, }"] [2025/04/04 15:41:06.385 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 306, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:46, a: 306, }"] [row2="{ b: 306, c: yiSmIdagro, zFzkK: IsNull, d: 15:45:46, a: 306, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:06 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383 {"id":"9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752463} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecc86e5 9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383 /tidb/cdc/default/default/upstream/7489359750431305597 {"id":7489359750431305597,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383 {"id":"9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752463} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecc86e5 9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383 /tidb/cdc/default/default/upstream/7489359750431305597 {"id":7489359750431305597,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383 {"id":"9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752463} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbecc86e5 9c9dbb3e-5a2f-4da6-a69e-6f8fc7285383 /tidb/cdc/default/default/upstream/7489359750431305597 {"id":7489359750431305597,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint_check_ts.cli.14129.out cli changefeed create --start-ts=457114245284233218 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/syncpoint_check_ts/conf/changefeed.toml Starting TLS TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:06.486 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 307, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:47, a: 307, }"] [row2="{ a: 307, b: 307, c: FZ9JT, zFzkK: IsNull, d: 15:45:47, }"] [2025/04/04 15:41:06.586 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:45:48, a: 308, b: 308, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 308, b: 308, c: iVAq7gv, zFzkK: IsNull, d: 15:45:48, }"] Create changefeed successfully! ID: 73098666-ec38-4c38-97ac-52c63ed9c68d Info: {"upstream_id":7489359750431305597,"namespace":"default","id":"73098666-ec38-4c38-97ac-52c63ed9c68d","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:41:06.52587494+08:00","start_ts":457114245284233218,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114245284233218,"checkpoint_ts":457114245284233218,"checkpoint_time":"2025-04-04 15:41:01.564"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:06.686 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 309, b: 309, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:49, }"] [row2="{ a: 309, b: 309, c: LNaClEMwkC, zFzkK: IsNull, d: 15:45:49, }"] [2025/04/04 15:41:06.786 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:50, a: 310, b: 310, c: Nw9p, }"] [row2="{ a: 310, b: 310, c: ZcfItDbEo, zFzkK: IsNull, d: 15:45:50, }"] [2025/04/04 15:41:06.886 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 311, b: 311, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:51, }"] [row2="{ a: 311, b: 311, c: ltmEIRVzE, zFzkK: IsNull, d: 15:45:51, }"] table consistent_replicate_storage_file.t2 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x [2025/04/04 15:41:06.987 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ zFzkK: IsNull, d: 15:45:52, a: 312, b: 312, c: 1Twk, }"] [2025/04/04 15:41:07.087 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 313, b: 313, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:53, }"] [row2="{ zFzkK: IsNull, d: 15:45:53, a: 313, b: 313, c: PRRMKM9apa, }"] [2025/04/04 15:41:07.187 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:54, a: 314, b: 314, }"] [row2="{ a: 314, b: 314, c: A28LmrRuSh, zFzkK: IsNull, d: 15:45:54, }"] [2025/04/04 15:41:07.287 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 315, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:55, a: 315, }"] [row2="{ a: 315, b: 315, c: rKttn, zFzkK: IsNull, d: 15:45:55, }"] [2025/04/04 15:41:07.387 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ b: 316, c: wZYCKR6fL, zFzkK: IsNull, d: 15:45:56, a: 316, }"] [2025/04/04 15:41:07.488 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:45:57, a: 317, b: 317, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: fbqjWpsp, zFzkK: IsNull, d: 15:45:57, a: 317, b: 317, }"] [2025/04/04 15:41:07.588 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 318, b: 318, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:58, }"] [row2="{ a: 318, b: 318, c: Jpnp93X3bk, zFzkK: IsNull, d: 15:45:58, }"] [2025/04/04 15:41:07.688 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:45:59, a: 319, b: 319, }"] [row2="{ b: 319, c: p, zFzkK: IsNull, d: 15:45:59, a: 319, }"] Starting TLS TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:07.788 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 320, b: 320, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:00, }"] [row2="{ b: 320, c: vYD, zFzkK: IsNull, d: 15:46:00, a: 320, }"] [2025/04/04 15:41:07.888 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:01, a: 321, b: 321, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 321, b: 321, c: Hl, zFzkK: IsNull, d: 15:46:01, }"] start tidb cluster in /tmp/tidb_cdc_test/sql_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:07.989 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 322, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:02, a: 322, }"] [row2="{ d: 15:46:02, a: 322, b: 322, c: rnN, zFzkK: IsNull, }"] [2025/04/04 15:41:08.089 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 323, b: 323, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:03, }"] [row2="{ a: 323, b: 323, c: JgT7p6V, zFzkK: IsNull, d: 15:46:03, }"] [2025/04/04 15:41:08.189 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 324, b: 324, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:04, }"] [row2="{ zFzkK: IsNull, d: 15:46:04, a: 324, b: 324, c: cvrwk0h, }"] [2025/04/04 15:41:08.289 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 325, b: 325, c: zAf, zFzkK: IsNull, d: 15:46:05, }"] [2025/04/04 15:41:08.389 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 326, b: 326, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:06, }"] [row2="{ zFzkK: IsNull, d: 15:46:06, a: 326, b: 326, c: SzRIuc, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359795274163351 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc43480003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:10613, start at 2025-04-04 15:41:06.645154551 +0800 CST m=+2.379669605 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:06.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:06.642 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:06.642 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359795274163351 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc43480003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:10613, start at 2025-04-04 15:41:06.645154551 +0800 CST m=+2.379669605 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:06.653 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:06.642 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:06.642 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359794228650463 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc44bc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-5fgk4-h679d-7n89q, pid:10690, start at 2025-04-04 15:41:06.769914827 +0800 CST m=+2.423862975 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:06.778 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:06.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:06.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2025/04/04 15:41:08.490 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 327, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:07, a: 327, }"] [row2="{ zFzkK: IsNull, d: 15:46:07, a: 327, b: 327, c: gENlOuhsf, }"] [2025/04/04 15:41:08.590 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 328, b: 328, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:08, }"] [row2="{ c: gcqyjk, zFzkK: IsNull, d: 15:46:08, a: 328, b: 328, }"] [2025/04/04 15:41:08.690 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 329, b: 329, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:09, }"] [row2="{ a: 329, b: 329, c: xNT4l, zFzkK: IsNull, d: 15:46:09, }"] Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:41:08.790 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:11, a: 330, b: 330, }"] [row2="{ d: 15:46:11, a: 330, b: 330, c: 5MMUSj, zFzkK: IsNull, }"] [2025/04/04 15:41:08.890 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 331, b: 331, c: 8f, zFzkK: IsNull, d: 15:46:12, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:08 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4084c1f4-c6a4-4e1b-bd63-3320920b9f43 {"id":"4084c1f4-c6a4-4e1b-bd63-3320920b9f43","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752466} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbed8dce5 4084c1f4-c6a4-4e1b-bd63-3320920b9f43 /tidb/cdc/default/default/upstream/7489359764150449268 {"id":7489359764150449268,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4084c1f4-c6a4-4e1b-bd63-3320920b9f43 {"id":"4084c1f4-c6a4-4e1b-bd63-3320920b9f43","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752466} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbed8dce5 4084c1f4-c6a4-4e1b-bd63-3320920b9f43 /tidb/cdc/default/default/upstream/7489359764150449268 {"id":7489359764150449268,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4084c1f4-c6a4-4e1b-bd63-3320920b9f43 {"id":"4084c1f4-c6a4-4e1b-bd63-3320920b9f43","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752466} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbed8dce5 4084c1f4-c6a4-4e1b-bd63-3320920b9f43 /tidb/cdc/default/default/upstream/7489359764150449268 {"id":7489359764150449268,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114245981536257 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.14072.out cli changefeed create --start-ts=457114245981536257 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml [2025/04/04 15:41:08.990 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 332, b: 332, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:04, }"] [row2="{ a: 332, b: 332, c: 72k1zeF6I, zFzkK: IsNull, d: 15:48:04, }"] [2025/04/04 15:41:09.091 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 333, b: 333, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:14, }"] [row2="{ a: 333, b: 333, c: 6dQ0wPf, zFzkK: IsNull, d: 15:46:14, }"] [2025/04/04 15:41:09.191 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:15, a: 334, b: 334, }"] [row2="{ d: 15:46:15, a: 334, b: 334, c: o8xK9C3m, zFzkK: IsNull, }"] Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489359764150449268,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:41:09.173808495+08:00","start_ts":457114245981536257,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114245981536257,"checkpoint_ts":457114245981536257,"checkpoint_time":"2025-04-04 15:41:04.224"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:09.291 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 335, b: 335, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:16, }"] [row2="{ c: uVQ, zFzkK: IsNull, d: 15:46:16, a: 335, b: 335, }"] [2025/04/04 15:41:09.391 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:17, a: 336, b: 336, }"] [row2="{ a: 336, b: 336, c: 59pZjV, zFzkK: IsNull, d: 15:46:17, }"] table consistent_replicate_storage_file.t2 not exists for 5-th check, retry later [2025/04/04 15:41:09.491 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 337, b: 337, c: 8Z7IfHd, zFzkK: IsNull, d: 15:46:18, }"] [2025/04/04 15:41:09.592 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 338, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:19, a: 338, }"] [row2="{ d: 15:46:19, a: 338, b: 338, c: ZTo5, zFzkK: IsNull, }"] [2025/04/04 15:41:09.692 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:20, a: 339, b: 339, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:46:20, a: 339, b: 339, c: CyuNjK, }"] [2025/04/04 15:41:09.792 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 340, b: 340, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:21, }"] [row2="{ c: mEQzdVzi, zFzkK: IsNull, d: 15:46:21, a: 340, b: 340, }"] [2025/04/04 15:41:09.892 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:22, a: 341, b: 341, c: Nw9p, }"] [row2="{ b: 341, c: rBBDeSp8Z, zFzkK: IsNull, d: 15:46:22, a: 341, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:09.993 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:23, a: 342, b: 342, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 342, b: 342, c: Q8PtxLB, zFzkK: IsNull, d: 15:46:23, }"] [2025/04/04 15:41:10.093 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:24, a: 343, b: 343, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: 4oC, zFzkK: IsNull, d: 15:46:24, a: 343, b: 343, }"] [2025/04/04 15:41:10.193 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:25, a: 344, b: 344, }"] [row2="{ d: 15:46:25, a: 344, b: 344, c: Tzlp7aBf4q, zFzkK: IsNull, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359808814298390 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc6dac0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:37855, start at 2025-04-04 15:41:09.360230825 +0800 CST m=+1.496840252 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:09.367 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:09.355 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:09.355 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Fri Apr 4 15:41:10 CST 2025] <<<<<< START cdc server in http_api_tls_with_user_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + GO_FAILPOINTS= + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls_with_user_auth.3789837900.out server --log-file /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_http_api_tls_with_user_auth_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_data_http_api_tls_with_user_auth_tls1 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls_with_user_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114246308954114, "checkpoint_time": "2025-04-04 15:41:05.473", "error": null }' + echo '{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114246308954114, "checkpoint_time": "2025-04-04 15:41:05.473", "error": null }' { "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114246308954114, "checkpoint_time": "2025-04-04 15:41:05.473", "error": null } ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114246308954114, '"checkpoint_time":' '"2025-04-04' '15:41:05.473",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114246308954114, '"checkpoint_time":' '"2025-04-04' '15:41:05.473",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11672.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name [2025/04/04 15:41:10.293 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 345, b: 345, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:26, }"] [row2="{ b: 345, c: uZ, zFzkK: IsNull, d: 15:46:26, a: 345, }"] [2025/04/04 15:41:10.393 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 346, b: 346, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:27, }"] [row2="{ b: 346, c: HCRixP, zFzkK: IsNull, d: 15:46:27, a: 346, }"] + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later [2025/04/04 15:41:10.493 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 347, b: 347, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:28, }"] [row2="{ a: 347, b: 347, c: A1e3P, zFzkK: IsNull, d: 15:46:28, }"] [2025/04/04 15:41:10.594 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 348, b: 348, c: kK, zFzkK: IsNull, d: 15:46:29, }"] [2025/04/04 15:41:10.694 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 349, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:30, a: 349, }"] [row2="{ zFzkK: IsNull, d: 15:46:30, a: 349, b: 349, c: 127lqCesE, }"] table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7489359713796286682,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:40:58.228166085+08:00","start_ts":457114242835283969,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":0,"checkpoint_ts":457114246308954114,"checkpoint_time":"2025-04-04 15:41:05.473"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:10.794 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 350, b: 350, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:31, }"] [row2="{ a: 350, b: 350, c: 0vsFeP, zFzkK: IsNull, d: 15:46:31, }"] [2025/04/04 15:41:10.894 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 351, b: 351, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:32, }"] [row2="{ c: BMN7VoO, zFzkK: IsNull, d: 15:46:32, a: 351, b: 351, }"] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12000.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:41:10.994 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:33, a: 352, b: 352, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:46:33, a: 352, b: 352, c: 1qeYF6, }"] [2025/04/04 15:41:11.095 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:34, a: 353, b: 353, c: Nw9p, }"] [row2="{ a: 353, b: 353, c: NqMG, zFzkK: IsNull, d: 15:46:34, }"] [2025/04/04 15:41:11.195 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 354, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:35, a: 354, }"] [row2="{ zFzkK: IsNull, d: 15:46:35, a: 354, b: 354, c: VDs, }"] table consistent_replicate_storage_file.t2 exists [2025/04/04 15:41:11.295 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 355, b: 355, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:36, }"] [row2="{ a: 355, b: 355, c: y, zFzkK: IsNull, d: 15:46:36, }"] [2025/04/04 15:41:11.395 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 356, b: 356, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:37, }"] [row2="{ a: 356, b: 356, c: s, zFzkK: IsNull, d: 15:46:37, }"] [2025/04/04 15:41:11.495 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:38, a: 357, b: 357, c: Nw9p, }"] [row2="{ b: 357, c: PW, zFzkK: IsNull, d: 15:46:38, a: 357, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:11.595 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 358, b: 358, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:39, }"] [row2="{ a: 358, b: 358, c: 7ZR, zFzkK: IsNull, d: 15:46:39, }"] [2025/04/04 15:41:11.696 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:40, a: 359, b: 359, }"] [row2="{ b: 359, c: 2Sx8Z62WI, zFzkK: IsNull, d: 15:46:40, a: 359, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359806631994089 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc77b40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:22281, start at 2025-04-04 15:41:10.045758952 +0800 CST m=+2.471376885 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:10.063 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:10.047 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:10.047 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359806631994089 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc77b40007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:22281, start at 2025-04-04 15:41:10.045758952 +0800 CST m=+2.471376885 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:10.063 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:10.047 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:10.047 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359801261732565 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc7a8c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:22342, start at 2025-04-04 15:41:10.198983118 +0800 CST m=+2.536328837 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:10.207 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:10.179 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:10.179 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check diff successfully [2025/04/04 15:41:11.796 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:41, a: 360, b: 360, }"] [row2="{ a: 360, b: 360, c: JHapNizt, zFzkK: IsNull, d: 15:46:41, }"] [2025/04/04 15:41:11.896 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 361, b: 361, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:42, }"] [row2="{ a: 361, b: 361, c: M8MKR, zFzkK: IsNull, d: 15:46:42, }"] [2025/04/04 15:41:11.996 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 362, b: 362, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:43, }"] [row2="{ a: 362, b: 362, c: nfWp, zFzkK: IsNull, d: 15:46:43, }"] Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... [2025/04/04 15:41:12.096 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:44, a: 363, b: 363, c: Nw9p, }"] [row2="{ a: 363, b: 363, c: iQJ7hdc, zFzkK: IsNull, d: 15:46:44, }"] [2025/04/04 15:41:12.196 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 364, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:45, a: 364, }"] [row2="{ c: Mr8TE3, zFzkK: IsNull, d: 15:46:45, a: 364, b: 364, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x [2025/04/04 15:41:12.297 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:46, a: 365, b: 365, c: Nw9p, }"] [row2="{ a: 365, b: 365, c: Bxbp, zFzkK: IsNull, d: 15:46:46, }"] [2025/04/04 15:41:12.397 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:47, a: 366, b: 366, }"] [row2="{ a: 366, b: 366, c: lfDkAcD6id, zFzkK: IsNull, d: 15:46:47, }"] [2025/04/04 15:41:12.497 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:48, a: 367, b: 367, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ d: 15:46:48, a: 367, b: 367, c: Spx7c5, zFzkK: IsNull, }"] nonempty select id, val from test.availability1 where id=3 and val=3 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11709.out cli changefeed --changefeed-id custom-changefeed-name resume + set +x + tso='457114247755726849 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114247755726849 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:41:12 CST 2025] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1204312045.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server1 --cluster-id test1 --addr 127.0.0.1:8300 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:12.597 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 368, b: 368, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:49, }"] [row2="{ a: 368, b: 368, c: J19, zFzkK: IsNull, d: 15:46:49, }"] [2025/04/04 15:41:12.697 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 369, b: 369, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:50, }"] [row2="{ zFzkK: IsNull, d: 15:46:50, a: 369, b: 369, c: EWtnLf0aIR, }"] wait process cdc.test exit for 2-th time... table ddl_attributes.attributes_t1_new not exists for 3-th check, retry later [2025/04/04 15:41:12.798 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 370, b: 370, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:51, }"] [row2="{ b: 370, c: RePyvNe, zFzkK: IsNull, d: 15:46:51, a: 370, }"] [2025/04/04 15:41:12.898 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 371, b: 371, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:52, }"] [row2="{ a: 371, b: 371, c: B, zFzkK: IsNull, d: 15:46:52, }"] [2025/04/04 15:41:12.998 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 372, b: 372, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:53, }"] [row2="{ c: i5IcDX4iR, zFzkK: IsNull, d: 15:46:53, a: 372, b: 372, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_expire_capture [Fri Apr 4 15:41:12 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.62206222.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_capture.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table test.t1 exists + sleep 5 [2025/04/04 15:41:13.098 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:54, a: 373, b: 373, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ b: 373, c: D8zoiD4ZLL, zFzkK: IsNull, d: 15:46:54, a: 373, }"] [2025/04/04 15:41:13.198 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 374, b: 374, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:55, }"] [row2="{ c: V, zFzkK: IsNull, d: 15:46:55, a: 374, b: 374, }"] wait process cdc.test exit for 3-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:13.298 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:56, a: 375, b: 375, c: Nw9p, }"] [row2="{ b: 375, c: c, zFzkK: IsNull, d: 15:46:56, a: 375, }"] [2025/04/04 15:41:13.399 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:57, a: 376, b: 376, }"] [row2="{ a: 376, b: 376, c: a4XEuPg4ID, zFzkK: IsNull, d: 15:46:57, }"] [2025/04/04 15:41:13.499 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:46:58, a: 377, b: 377, }"] [row2="{ b: 377, c: gpkp, zFzkK: IsNull, d: 15:46:58, a: 377, }"] + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:13 GMT < Content-Length: 1210 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29358585-ef28-4d9a-a7c1-9f488c358ec2 {"id":"29358585-ef28-4d9a-a7c1-9f488c358ec2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752470} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf0389dd 29358585-ef28-4d9a-a7c1-9f488c358ec2 /tidb/cdc/default/default/upstream/7489359808814298390 {"id":7489359808814298390,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29358585-ef28-4d9a-a7c1-9f488c358ec2 {"id":"29358585-ef28-4d9a-a7c1-9f488c358ec2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752470} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf0389dd 29358585-ef28-4d9a-a7c1-9f488c358ec2 /tidb/cdc/default/default/upstream/7489359808814298390 {"id":7489359808814298390,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29358585-ef28-4d9a-a7c1-9f488c358ec2 {"id":"29358585-ef28-4d9a-a7c1-9f488c358ec2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752470} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf0389dd 29358585-ef28-4d9a-a7c1-9f488c358ec2 /tidb/cdc/default/default/upstream/7489359808814298390 {"id":7489359808814298390,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:41:13.599 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:46:59, a: 378, b: 378, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 378, b: 378, c: 7, zFzkK: IsNull, d: 15:46:59, }"] [2025/04/04 15:41:13.699 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 379, b: 379, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:00, }"] [row2="{ a: 379, b: 379, c: 6, zFzkK: IsNull, d: 15:47:00, }"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:41:13 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1135711359.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:41:13.799 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:47:01, a: 380, b: 380, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ b: 380, c: PoLwXE7, zFzkK: IsNull, d: 15:47:01, a: 380, }"] [2025/04/04 15:41:13.899 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 381, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:02, a: 381, }"] [row2="{ b: 381, c: GB9, zFzkK: IsNull, d: 15:47:02, a: 381, }"] [2025/04/04 15:41:14.000 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:03, a: 382, b: 382, }"] [row2="{ d: 15:47:03, a: 382, b: 382, c: LvoLFiCb, zFzkK: IsNull, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.23617.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:41:14.100 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 383, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:04, a: 383, }"] [row2="{ b: 383, c: fwh6l, zFzkK: IsNull, d: 15:47:04, a: 383, }"] [2025/04/04 15:41:14.200 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 384, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:05, a: 384, }"] [row2="{ b: 384, c: L2S4l, zFzkK: IsNull, d: 15:47:05, a: 384, }"] + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/owner_remove_table_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:41:14.300 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 385, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:06, a: 385, }"] [row2="{ d: 15:47:06, a: 385, b: 385, c: hbQ2, zFzkK: IsNull, }"] [2025/04/04 15:41:14.400 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 386, b: 386, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:07, }"] [row2="{ c: gIC, zFzkK: IsNull, d: 15:47:07, a: 386, b: 386, }"] [2025/04/04 15:41:14.501 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 387, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:08, a: 387, }"] [row2="{ a: 387, b: 387, c: CxW37I1n, zFzkK: IsNull, d: 15:47:08, }"] table ddl_attributes.attributes_t1_new not exists for 4-th check, retry later [2025/04/04 15:41:14.601 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 388, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:09, a: 388, }"] [row2="{ d: 15:47:09, a: 388, b: 388, c: u85yGe, zFzkK: IsNull, }"] [2025/04/04 15:41:14.701 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 389, b: 389, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:10, }"] [row2="{ a: 389, b: 389, c: S, zFzkK: IsNull, d: 15:47:10, }"] [2025/04/04 15:41:14.801 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:12, a: 390, b: 390, }"] [row2="{ d: 15:47:12, a: 390, b: 390, c: qivoYP, zFzkK: IsNull, }"] [2025/04/04 15:41:14.901 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:13, a: 391, b: 391, }"] [row2="{ a: 391, b: 391, c: HK3, zFzkK: IsNull, d: 15:47:13, }"] [2025/04/04 15:41:15.002 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:14, a: 392, b: 392, }"] [row2="{ a: 392, b: 392, c: 7, zFzkK: IsNull, d: 15:47:14, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:15.102 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 393, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:15, a: 393, }"] [row2="{ d: 15:47:15, a: 393, b: 393, c: K4, zFzkK: IsNull, }"] [2025/04/04 15:41:15.202 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 394, b: 394, c: 00hfzSOnb, zFzkK: IsNull, d: 15:47:16, }"] [2025/04/04 15:41:15.302 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:17, a: 395, b: 395, c: Nw9p, }"] [row2="{ d: 15:47:17, a: 395, b: 395, c: WqUz2, zFzkK: IsNull, }"] [Fri Apr 4 15:41:15 CST 2025] <<<<<< START cdc server in http_api_tls_with_user_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls_with_user_auth.3796237964.out server --log-file /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_http_api_tls_with_user_auth_tls2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls_with_user_auth/cdc_data_http_api_tls_with_user_auth_tls2 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls_with_user_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8301 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:15.402 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 396, b: 396, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:18, }"] [row2="{ a: 396, b: 396, c: xMFABMPCRX, zFzkK: IsNull, d: 15:47:18, }"] [2025/04/04 15:41:15.502 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 397, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:19, a: 397, }"] [row2="{ a: 397, b: 397, c: C, zFzkK: IsNull, d: 15:47:19, }"] [2025/04/04 15:41:15.603 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 398, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:20, a: 398, }"] [row2="{ zFzkK: IsNull, d: 15:47:20, a: 398, b: 398, c: xV, }"] [2025/04/04 15:41:15.703 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 399, b: 399, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:21, }"] [row2="{ a: 399, b: 399, c: rsW9NK, zFzkK: IsNull, d: 15:47:21, }"] [2025/04/04 15:41:15.803 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 400, b: 400, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:23, }"] [row2="{ a: 400, b: 400, c: mhnP2dqPg5, zFzkK: IsNull, d: 15:47:23, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:15 GMT < Content-Length: 802 < Content-Type: text/plain; charset=utf-8 < { [802 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/001e5d4f-7c91-4304-aef0-3aa87d59ed10 {"id":"001e5d4f-7c91-4304-aef0-3aa87d59ed10","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752472} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/223195ffbef766f3 001e5d4f-7c91-4304-aef0-3aa87d59ed10 /tidb/cdc/test1/default/upstream/7489359795274163351 {"id":7489359795274163351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/001e5d4f-7c91-4304-aef0-3aa87d59ed10 {"id":"001e5d4f-7c91-4304-aef0-3aa87d59ed10","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752472} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/223195ffbef766f3 001e5d4f-7c91-4304-aef0-3aa87d59ed10 /tidb/cdc/test1/default/upstream/7489359795274163351 {"id":7489359795274163351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/001e5d4f-7c91-4304-aef0-3aa87d59ed10 {"id":"001e5d4f-7c91-4304-aef0-3aa87d59ed10","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752472} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/223195ffbef766f3 001e5d4f-7c91-4304-aef0-3aa87d59ed10 /tidb/cdc/test1/default/upstream/7489359795274163351 {"id":7489359795274163351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:41:15 CST 2025] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1210112103.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server2 --cluster-id test2 --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359462573592486 TiDB Cluster ID. tikv_gc_leader_uuid 657fef799a80007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9074, start at 2025-04-04 15:39:50.262647483 +0800 CST m=+2.333469047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.270 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.250 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.250 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359467753540872 TiDB Cluster ID. tikv_gc_leader_uuid 657fef79994000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-n8g2x-37nt4-vn2cl, pid:9146, start at 2025-04-04 15:39:50.273178944 +0800 CST m=+2.246077329 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:41:50.281 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 60m All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:39:50.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:29:50.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'normal'@'%' start tidb cluster failed + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 4000 + run_sql 'set global sql_mode='\''ANSI_QUOTES'\'';' 127.0.0.1 3306 [2025/04/04 15:41:15.903 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:24, a: 401, b: 401, }"] [row2="{ a: 401, b: 401, c: RKEatgYzUq, zFzkK: IsNull, d: 15:47:24, }"] [2025/04/04 15:41:16.003 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 402, b: 402, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:25, }"] [row2="{ a: 402, b: 402, c: G, zFzkK: IsNull, d: 15:47:25, }"] + set +x + tso='457114248635482113 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114248635482113 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14327.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:16 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f71680 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114247794786306 MinTableBarrierTs:457114248843362306 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing [2025/04/04 15:41:16.104 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:47:26, a: 403, b: 403, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 403, b: 403, c: SQ, zFzkK: IsNull, d: 15:47:26, }"] span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing [2025/04/04 15:41:16.204 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 404, b: 404, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:27, }"] [row2="{ a: 404, b: 404, c: FDh9FEdl94, zFzkK: IsNull, d: 15:47:27, }"] span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/321be9f2-1c6b-4d3f-8492-478748f9e51d [2025/04/04 15:41:16.304 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 405, b: 405, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:28, }"] [row2="{ a: 405, b: 405, c: Cv, zFzkK: IsNull, d: 15:47:28, }"] {"id":"321be9f2-1c6b-4d3f-8492-478748f9e51d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752473} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe66b 321be9f2-1c6b-4d3f-8492-478748f9e51d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114247794786306,"min-table-barrier-ts":457114248843362306,"admin-job-type":0} /tidb/cdc/default/default/task/position/321be9f2-1c6b-4d3f-8492-478748f9e51d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f71680 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114247794786306 MinTableBarrierTs:457114248843362306 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/321be9f2-1c6b-4d3f-8492-478748f9e51d {"id":"321be9f2-1c6b-4d3f-8492-478748f9e51d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752473} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe66b 321be9f2-1c6b-4d3f-8492-478748f9e51d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114247794786306,"min-table-barrier-ts":457114248843362306,"admin-job-type":0} /tidb/cdc/default/default/task/position/321be9f2-1c6b-4d3f-8492-478748f9e51d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003f71680 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114247794786306 MinTableBarrierTs:457114248843362306 AdminJobType:noop} span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114247794786306, checkpointTs: 457114247794786306, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/321be9f2-1c6b-4d3f-8492-478748f9e51d {"id":"321be9f2-1c6b-4d3f-8492-478748f9e51d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752473} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe66b 321be9f2-1c6b-4d3f-8492-478748f9e51d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114247794786306,"min-table-barrier-ts":457114248843362306,"admin-job-type":0} /tidb/cdc/default/default/task/position/321be9f2-1c6b-4d3f-8492-478748f9e51d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2025/04/04 15:41:16.404 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 406, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:29, a: 406, }"] [row2="{ a: 406, b: 406, c: WnIuQII, zFzkK: IsNull, d: 15:47:29, }"] [2025/04/04 15:41:16.504 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:30, a: 407, b: 407, }"] [row2="{ a: 407, b: 407, c: ahYBP, zFzkK: IsNull, d: 15:47:30, }"] [2025/04/04 15:41:16.604 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:47:31, a: 408, b: 408, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ d: 15:47:31, a: 408, b: 408, c: LXUb, zFzkK: IsNull, }"] [2025/04/04 15:41:16.705 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:32, a: 409, b: 409, }"] [row2="{ c: K, zFzkK: IsNull, d: 15:47:32, a: 409, b: 409, }"] [2025/04/04 15:41:16.805 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:33, a: 410, b: 410, }"] [row2="{ a: 410, b: 410, c: 8ForS, zFzkK: IsNull, d: 15:47:33, }"] ***************** properties ***************** "scanproportion"="0" "threadcount"="10" "insertproportion"="0" "readallfields"="true" "recordcount"="10000" "workload"="core" "mysql.db"="move_table" "updateproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" "mysql.port"="4000" "operationcount"="0" "readproportion"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" ********************************************** [2025/04/04 15:41:16.905 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 411, b: 411, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:34, }"] [row2="{ c: 8JE9bWC, zFzkK: IsNull, d: 15:47:34, a: 411, b: 411, }"] [2025/04/04 15:41:17.005 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 412, b: 412, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:35, }"] [row2="{ a: 412, b: 412, c: 8, zFzkK: IsNull, d: 15:47:35, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:16 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {UpstreamID:7489359564073779471 Namespace:default ID:7fe74488-ff74-4a3f-8d0c-cd20de57ea8e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:20.045886367 +0800 CST StartTs:457114234376421381 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d6e120 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114234389528585} {CheckpointTs:457114248951627780 MinTableBarrierTs:457114248951627781 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114248951627782, checkpointTs: 457114248951627780, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114248951627780, checkpointTs: 457114248951627780, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/563b9f1b-62c2-423b-a164-46b77d8de469 {"id":"563b9f1b-62c2-423b-a164-46b77d8de469","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752474} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe26df10 563b9f1b-62c2-423b-a164-46b77d8de469 /tidb/cdc/default/default/changefeed/info/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"upstream-id":7489359564073779471,"namespace":"default","changefeed-id":"7fe74488-ff74-4a3f-8d0c-cd20de57ea8e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:20.045886367+08:00","start-ts":457114234376421381,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114234389528585} /tidb/cdc/default/default/changefeed/status/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"checkpoint-ts":457114248951627780,"min-table-barrier-ts":457114248951627781,"admin-job-type":0} /tidb/cdc/default/default/task/position/563b9f1b-62c2-423b-a164-46b77d8de469/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359564073779471 {"id":7489359564073779471,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {UpstreamID:7489359564073779471 Namespace:default ID:7fe74488-ff74-4a3f-8d0c-cd20de57ea8e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:20.045886367 +0800 CST StartTs:457114234376421381 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d6e120 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114234389528585} {CheckpointTs:457114248951627780 MinTableBarrierTs:457114248951627781 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114248951627782, checkpointTs: 457114248951627780, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114248951627780, checkpointTs: 457114248951627780, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/563b9f1b-62c2-423b-a164-46b77d8de469 {"id":"563b9f1b-62c2-423b-a164-46b77d8de469","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752474} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe26df10 563b9f1b-62c2-423b-a164-46b77d8de469 /tidb/cdc/default/default/changefeed/info/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"upstream-id":7489359564073779471,"namespace":"default","changefeed-id":"7fe74488-ff74-4a3f-8d0c-cd20de57ea8e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:20.045886367+08:00","start-ts":457114234376421381,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114234389528585} /tidb/cdc/default/default/changefeed/status/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"checkpoint-ts":457114248951627780,"min-table-barrier-ts":457114248951627781,"admin-job-type":0} /tidb/cdc/default/default/task/position/563b9f1b-62c2-423b-a164-46b77d8de469/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359564073779471 {"id":7489359564073779471,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {UpstreamID:7489359564073779471 Namespace:default ID:7fe74488-ff74-4a3f-8d0c-cd20de57ea8e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:40:20.045886367 +0800 CST StartTs:457114234376421381 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d6e120 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114234389528585} {CheckpointTs:457114248951627780 MinTableBarrierTs:457114248951627781 AdminJobType:noop} span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114248951627782, checkpointTs: 457114248951627780, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114248951627780, checkpointTs: 457114248951627780, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114248951627781, checkpointTs: 457114248951627780, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/563b9f1b-62c2-423b-a164-46b77d8de469 {"id":"563b9f1b-62c2-423b-a164-46b77d8de469","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752474} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbe26df10 563b9f1b-62c2-423b-a164-46b77d8de469 /tidb/cdc/default/default/changefeed/info/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"upstream-id":7489359564073779471,"namespace":"default","changefeed-id":"7fe74488-ff74-4a3f-8d0c-cd20de57ea8e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:40:20.045886367+08:00","start-ts":457114234376421381,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114234389528585} /tidb/cdc/default/default/changefeed/status/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"checkpoint-ts":457114248951627780,"min-table-barrier-ts":457114248951627781,"admin-job-type":0} /tidb/cdc/default/default/task/position/563b9f1b-62c2-423b-a164-46b77d8de469/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359564073779471 {"id":7489359564073779471,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later [2025/04/04 15:41:17.105 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 413, b: 413, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:36, }"] [row2="{ a: 413, b: 413, c: IZlQFK5, zFzkK: IsNull, d: 15:47:36, }"] [2025/04/04 15:41:17.206 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 414, b: 414, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:37, }"] [row2="{ a: 414, b: 414, c: 4ECsGlV, zFzkK: IsNull, d: 15:47:37, }"] [2025/04/04 15:41:17.306 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 415, b: 415, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:38, }"] [row2="{ d: 15:47:38, a: 415, b: 415, c: Jd73NxX, zFzkK: IsNull, }"] [2025/04/04 15:41:17.406 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 416, b: 416, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:39, }"] [row2="{ b: 416, c: dUoj9BAp, zFzkK: IsNull, d: 15:47:39, a: 416, }"] [2025/04/04 15:41:17.506 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ d: 15:47:40, a: 417, b: 417, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: u02e, zFzkK: IsNull, d: 15:47:40, a: 417, b: 417, }"] "is-owner": true, run task successfully + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114249192538116, "checkpoint_time": "2025-04-04 15:41:16.473", "error": null }' + echo '{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114249192538116, "checkpoint_time": "2025-04-04 15:41:16.473", "error": null }' { "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114249192538116, "checkpoint_time": "2025-04-04 15:41:16.473", "error": null } ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114249192538116, '"checkpoint_time":' '"2025-04-04' '15:41:16.473",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114249192538116, '"checkpoint_time":' '"2025-04-04' '15:41:16.473",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11783.out cli changefeed --changefeed-id custom-changefeed-name remove start tidb cluster in /tmp/tidb_cdc_test/owner_remove_table_error Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... [2025/04/04 15:41:17.606 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 418, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:41, a: 418, }"] [row2="{ d: 15:47:41, a: 418, b: 418, c: Vmu, zFzkK: IsNull, }"] [2025/04/04 15:41:17.707 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 419, b: 419, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:42, }"] [row2="{ b: 419, c: G, zFzkK: IsNull, d: 15:47:42, a: 419, }"] [2025/04/04 15:41:17.807 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 420, b: 420, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:43, }"] [row2="{ a: 420, b: 420, c: j6zj, zFzkK: IsNull, d: 15:47:43, }"] Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:41:17.907 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:44, a: 421, b: 421, c: Nw9p, }"] [row2="{ c: 7G7EiCxN, zFzkK: IsNull, d: 15:47:44, a: 421, b: 421, }"] [2025/04/04 15:41:18.007 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 422, b: 422, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:45, }"] [row2="{ zFzkK: IsNull, d: 15:47:45, a: 422, b: 422, c: MmN, }"] + set +x + tso='457114249178382337 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114249178382337 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114249178382337 + run_cdc_server --workdir /tmp/tidb_cdc_test/sql_mode --binary cdc.test [Fri Apr 4 15:41:17 CST 2025] <<<<<< START cdc server in sql_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.1435014352.out server --log-file /tmp/tidb_cdc_test/sql_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sql_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:18.107 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:47:46, a: 423, b: 423, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 423, b: 423, c: LhdId4LKRz, zFzkK: IsNull, d: 15:47:46, }"] [2025/04/04 15:41:18.208 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 424, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:35, a: 424, }"] [row2="{ d: 15:47:47, a: 424, b: 424, c: C7u, zFzkK: IsNull, }"] [2025/04/04 15:41:18.308 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 425, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:48, a: 425, }"] [row2="{ a: 425, b: 425, c: nhbQek6lXz, zFzkK: IsNull, d: 15:47:48, }"] + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 11997 22.9 0.6 4845048 2467928 ? Sl 15:40 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv1 jenkins 11998 32.0 0.6 4894712 2534140 ? Sl 15:40 0:07 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv2 jenkins 11999 22.6 0.6 4847600 2482940 ? Sl 15:40 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv3 jenkins 12001 31.0 0.6 4881396 2517060 ? Sl 15:40 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status_with_redo/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status_with_redo/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2793 0 --:--:-- --:--:-- --:--:-- 2825 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:41:16.523","puller_resolved_ts":"2025-04-04 15:41:10.623","last_synced_ts":"2025-04-04 15:41:10.623","now_ts":"2025-04-04 15:41:17.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:41:16.523","puller_resolved_ts":"2025-04-04' '15:41:10.623","last_synced_ts":"2025-04-04' '15:41:10.623","now_ts":"2025-04-04' '15:41:17.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:41:16.523","puller_resolved_ts":"2025-04-04' '15:41:10.623","last_synced_ts":"2025-04-04' '15:41:10.623","now_ts":"2025-04-04' '15:41:17.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 [2025/04/04 15:41:18.408 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:49, a: 426, b: 426, }"] [row2="{ d: 15:47:49, a: 426, b: 426, c: oyO, zFzkK: IsNull, }"] [2025/04/04 15:41:18.508 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 427, b: 427, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:50, }"] [row2="{ b: 427, c: FzaDO24qIu, zFzkK: IsNull, d: 15:47:50, a: 427, }"] Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 457114249454419973 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/ PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8301/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:18 GMT < Content-Length: 1661 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29358585-ef28-4d9a-a7c1-9f488c358ec2 {"id":"29358585-ef28-4d9a-a7c1-9f488c358ec2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752470} /tidb/cdc/default/__cdc_meta__/capture/cceea585-2486-4ad3-a1ed-f270b3520bb4 {"id":"cceea585-2486-4ad3-a1ed-f270b3520bb4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752475} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf0389dd 29358585-ef28-4d9a-a7c1-9f488c358ec2 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf038a1c cceea585-2486-4ad3-a1ed-f270b3520bb4 /tidb/cdc/default/default/upstream/7489359808814298390 {"id":7489359808814298390,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29358585-ef28-4d9a-a7c1-9f488c358ec2 {"id":"29358585-ef28-4d9a-a7c1-9f488c358ec2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752470} /tidb/cdc/default/__cdc_meta__/capture/cceea585-2486-4ad3-a1ed-f270b3520bb4 {"id":"cceea585-2486-4ad3-a1ed-f270b3520bb4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752475} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf0389dd 29358585-ef28-4d9a-a7c1-9f488c358ec2 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf038a1c cceea585-2486-4ad3-a1ed-f270b3520bb4 /tidb/cdc/default/default/upstream/7489359808814298390 {"id":7489359808814298390,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29358585-ef28-4d9a-a7c1-9f488c358ec2 {"id":"29358585-ef28-4d9a-a7c1-9f488c358ec2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752470} /tidb/cdc/default/__cdc_meta__/capture/cceea585-2486-4ad3-a1ed-f270b3520bb4 {"id":"cceea585-2486-4ad3-a1ed-f270b3520bb4","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752475} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf0389dd 29358585-ef28-4d9a-a7c1-9f488c358ec2 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffbf038a1c cceea585-2486-4ad3-a1ed-f270b3520bb4 /tidb/cdc/default/default/upstream/7489359808814298390 {"id":7489359808814298390,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:41:18.608 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 428, b: 428, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:51, }"] [row2="{ a: 428, b: 428, c: pdpBDDqO, zFzkK: IsNull, d: 15:47:51, }"] [2025/04/04 15:41:18.708 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ zFzkK: IsNull, d: 15:47:52, a: 429, b: 429, c: yQfo3n, }"] [2025/04/04 15:41:18.809 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 430, b: 430, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:53, }"] [row2="{ d: 15:47:53, a: 430, b: 430, c: lkUuu, zFzkK: IsNull, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) owner pid: 6225 owner id 321be9f2-1c6b-4d3f-8492-478748f9e51d process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v '321be9f2-1c6b-4d3f-8492-478748f9e51d' table ddl_attributes.finish_mark not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:18 GMT < Content-Length: 802 < Content-Type: text/plain; charset=utf-8 < { [802 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/251f776c-1112-4669-a55a-a58a42817b65 {"id":"251f776c-1112-4669-a55a-a58a42817b65","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752475} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/223195ffbef7671e 251f776c-1112-4669-a55a-a58a42817b65 /tidb/cdc/test2/default/upstream/7489359795274163351 {"id":7489359795274163351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/251f776c-1112-4669-a55a-a58a42817b65 {"id":"251f776c-1112-4669-a55a-a58a42817b65","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752475} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/223195ffbef7671e 251f776c-1112-4669-a55a-a58a42817b65 /tidb/cdc/test2/default/upstream/7489359795274163351 {"id":7489359795274163351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/251f776c-1112-4669-a55a-a58a42817b65 {"id":"251f776c-1112-4669-a55a-a58a42817b65","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752475} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/223195ffbef7671e 251f776c-1112-4669-a55a-a58a42817b65 /tidb/cdc/test2/default/upstream/7489359795274163351 {"id":7489359795274163351,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12156.out cli changefeed create --start-ts=457114247755726849 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed1.toml [2025/04/04 15:41:18.909 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:54, a: 431, b: 431, }"] [row2="{ c: z1q, zFzkK: IsNull, d: 15:47:54, a: 431, b: 431, }"] [2025/04/04 15:41:19.009 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 432, b: 432, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:55, }"] [row2="{ a: 432, b: 432, c: Iy, zFzkK: IsNull, d: 15:47:55, }"] /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe66b run task successfully process status: S nonempty select id, val from test.availability1 where id=2 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Create changefeed successfully! ID: f92ea2d3-fcd8-4585-ae54-be3e659a3770 Info: {"upstream_id":7489359795274163351,"namespace":"default","id":"f92ea2d3-fcd8-4585-ae54-be3e659a3770","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:19.100020437+08:00","start_ts":457114247755726849,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc1"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114247755726849,"checkpoint_ts":457114247755726849,"checkpoint_time":"2025-04-04 15:41:10.992"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:19.109 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:56, a: 433, b: 433, }"] [row2="{ a: 433, b: 433, c: 8n, zFzkK: IsNull, d: 15:47:56, }"] [2025/04/04 15:41:19.209 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 434, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:57, a: 434, }"] [row2="{ a: 434, b: 434, c: Ih54vqJ, zFzkK: IsNull, d: 15:47:57, }"] [2025/04/04 15:41:19.310 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:58, a: 435, b: 435, c: Nw9p, }"] [row2="{ a: 435, b: 435, c: HDgzebp, zFzkK: IsNull, d: 15:47:58, }"] Verifying downstream PD is started... Run finished, takes 2.527009002s INSERT - Takes(s): 2.5, Count: 10000, OPS: 3968.1, Avg(us): 2476, Min(us): 894, Max(us): 54247, 95th(us): 3000, 99th(us): 43000 [Fri Apr 4 15:41:19 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2370823710.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:19.410 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:47:59, a: 436, b: 436, }"] [row2="{ a: 436, b: 436, c: P, zFzkK: IsNull, d: 15:47:59, }"] [2025/04/04 15:41:19.510 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 437, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:00, a: 437, }"] [row2="{ d: 15:48:00, a: 437, b: 437, c: 3A, zFzkK: IsNull, }"] [2025/04/04 15:41:19.611 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 438, b: 438, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:01, }"] [row2="{ c: OnCCkTJEL, zFzkK: IsNull, d: 15:48:01, a: 438, b: 438, }"] [2025/04/04 15:41:19.711 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 439, b: 439, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:02, }"] [row2="{ a: 439, b: 439, c: 3pyOsJoB0, zFzkK: IsNull, d: 15:48:02, }"] [2025/04/04 15:41:19.811 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 440, b: 440, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:05, }"] [row2="{ b: 440, c: k2m, zFzkK: IsNull, d: 15:48:05, a: 440, }"] + set +x [2025/04/04 15:41:19.911 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 441, b: 441, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:06, }"] [row2="{ a: 441, b: 441, c: l0mCX, zFzkK: IsNull, d: 15:48:06, }"] [2025/04/04 15:41:20.011 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 442, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:07, a: 442, }"] [row2="{ b: 442, c: up, zFzkK: IsNull, d: 15:48:07, a: 442, }"] [2025/04/04 15:41:20.112 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:08, a: 443, b: 443, c: Nw9p, }"] [row2="{ c: kzsaLdZ, zFzkK: IsNull, d: 15:48:08, a: 443, b: 443, }"] [2025/04/04 15:41:20.212 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 444, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:09, a: 444, }"] [row2="{ a: 444, b: 444, c: 2L5C3RDm8, zFzkK: IsNull, d: 15:48:09, }"] [2025/04/04 15:41:20.312 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 445, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:10, a: 445, }"] [row2="{ a: 445, b: 445, c: uC, zFzkK: IsNull, d: 15:48:10, }"] [2025/04/04 15:41:20.412 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:11, a: 446, b: 446, }"] [row2="{ a: 446, b: 446, c: QUaGPKJ, zFzkK: IsNull, d: 15:48:11, }"] [2025/04/04 15:41:20.512 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:12, a: 447, b: 447, }"] [row2="{ c: R, zFzkK: IsNull, d: 15:48:12, a: 447, b: 447, }"] [2025/04/04 15:41:20.612 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 448, b: 448, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:13, }"] [row2="{ a: 448, b: 448, c: URVBs34H5, zFzkK: IsNull, d: 15:48:13, }"] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12201.out cli changefeed create --start-ts=457114247755726849 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8301 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed2.toml pass test: check health pass test: get status [2025/04/04 15:41:20.713 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:14, a: 449, b: 449, }"] [row2="{ b: 449, c: CNaY, zFzkK: IsNull, d: 15:48:14, a: 449, }"] [2025/04/04 15:41:20.813 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 450, b: 450, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:15, }"] [row2="{ a: 450, b: 450, c: Vrr, zFzkK: IsNull, d: 15:48:15, }"] table ddl_attributes.finish_mark not exists for 3-th check, retry later [2025/04/04 15:41:20.913 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:16, a: 451, b: 451, c: Nw9p, }"] [row2="{ zFzkK: IsNull, d: 15:48:16, a: 451, b: 451, c: YrTZf2Z, }"] [2025/04/04 15:41:21.013 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 452, b: 452, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:17, }"] [row2="{ zFzkK: IsNull, d: 15:48:17, a: 452, b: 452, c: b95Y5MMLq, }"] [2025/04/04 15:41:21.113 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:18, a: 453, b: 453, }"] [row2="{ c: vgVdXjYlW, zFzkK: IsNull, d: 15:48:18, a: 453, b: 453, }"] nonempty select id, val from test.availability1 where id=2 and val=22 Create changefeed successfully! ID: 49db6272-4fd9-4c38-ba84-2421f6c73f20 Info: {"upstream_id":7489359795274163351,"namespace":"default","id":"49db6272-4fd9-4c38-ba84-2421f6c73f20","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:20.946071778+08:00","start_ts":457114247755726849,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc2"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114247755726849,"checkpoint_ts":457114247755726849,"checkpoint_time":"2025-04-04 15:41:10.992"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:21.214 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 454, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:19, a: 454, }"] [row2="{ a: 454, b: 454, c: F1DRq, zFzkK: IsNull, d: 15:48:19, }"] [2025/04/04 15:41:21.314 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:48:20, a: 455, b: 455, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 455, b: 455, c: oP2, zFzkK: IsNull, d: 15:48:20, }"] run task successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:20 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489359462573592486 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:39:59.649976145 +0800 CST StartTs:457114227756498945 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038e5680 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229040742409} {CheckpointTs:457114232278220807 MinTableBarrierTs:457114232540364806 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f8e3d7a-b63e-4e70-96b2-ff6394772043 {"id":"7f8e3d7a-b63e-4e70-96b2-ff6394772043","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752396} /tidb/cdc/default/__cdc_meta__/capture/b753370d-d56f-452b-bde0-e44b4fe95d93 {"id":"b753370d-d56f-452b-bde0-e44b4fe95d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752478} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc755e8 7f8e3d7a-b63e-4e70-96b2-ff6394772043 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf12a8d3 b753370d-d56f-452b-bde0-e44b4fe95d93 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489359462573592486,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:39:59.649976145+08:00","start-ts":457114227756498945,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229040742409} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114232278220807,"min-table-barrier-ts":457114232540364806,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f8e3d7a-b63e-4e70-96b2-ff6394772043/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b753370d-d56f-452b-bde0-e44b4fe95d93/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359462573592486 {"id":7489359462573592486,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489359462573592486 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:39:59.649976145 +0800 CST StartTs:457114227756498945 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038e5680 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229040742409} {CheckpointTs:457114232278220807 MinTableBarrierTs:457114232540364806 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f8e3d7a-b63e-4e70-96b2-ff6394772043 {"id":"7f8e3d7a-b63e-4e70-96b2-ff6394772043","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752396} /tidb/cdc/default/__cdc_meta__/capture/b753370d-d56f-452b-bde0-e44b4fe95d93 {"id":"b753370d-d56f-452b-bde0-e44b4fe95d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752478} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc755e8 7f8e3d7a-b63e-4e70-96b2-ff6394772043 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf12a8d3 b753370d-d56f-452b-bde0-e44b4fe95d93 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489359462573592486,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:39:59.649976145+08:00","start-ts":457114227756498945,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229040742409} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114232278220807,"min-table-barrier-ts":457114232540364806,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f8e3d7a-b63e-4e70-96b2-ff6394772043/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b753370d-d56f-452b-bde0-e44b4fe95d93/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359462573592486 {"id":7489359462573592486,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/test-1 {UpstreamID:7489359462573592486 Namespace:default ID:test-1 SinkURI:mysql://root@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:39:59.649976145 +0800 CST StartTs:457114227756498945 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038e5680 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114229040742409} {CheckpointTs:457114232278220807 MinTableBarrierTs:457114232540364806 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7f8e3d7a-b63e-4e70-96b2-ff6394772043 {"id":"7f8e3d7a-b63e-4e70-96b2-ff6394772043","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752396} /tidb/cdc/default/__cdc_meta__/capture/b753370d-d56f-452b-bde0-e44b4fe95d93 {"id":"b753370d-d56f-452b-bde0-e44b4fe95d93","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752478} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbdc755e8 7f8e3d7a-b63e-4e70-96b2-ff6394772043 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf12a8d3 b753370d-d56f-452b-bde0-e44b4fe95d93 /tidb/cdc/default/default/changefeed/info/test-1 {"upstream-id":7489359462573592486,"namespace":"default","changefeed-id":"test-1","sink-uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:39:59.649976145+08:00","start-ts":457114227756498945,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114229040742409} /tidb/cdc/default/default/changefeed/status/test-1 {"checkpoint-ts":457114232278220807,"min-table-barrier-ts":457114232540364806,"admin-job-type":0} /tidb/cdc/default/default/task/position/7f8e3d7a-b63e-4e70-96b2-ff6394772043/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b753370d-d56f-452b-bde0-e44b4fe95d93/test-1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359462573592486 {"id":7489359462573592486,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114249178382337 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sql_mode.cli.14403.out cli changefeed create --start-ts=457114249178382337 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: create changefeed [2025/04/04 15:41:21.414 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 456, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:21, a: 456, }"] [row2="{ zFzkK: IsNull, d: 15:48:21, a: 456, b: 456, c: 03MQ, }"] [2025/04/04 15:41:21.514 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 457, b: 457, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:22, }"] [row2="{ b: 457, c: LHs3Z7AdT, zFzkK: IsNull, d: 15:48:22, a: 457, }"] [2025/04/04 15:41:21.614 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:23, a: 458, b: 458, }"] [row2="{ a: 458, b: 458, c: 2gtPo8r, zFzkK: IsNull, d: 15:48:23, }"] check_changefeed_state https://127.0.0.1:2579 changefeed-test1 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test1 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test1 -s + info='{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114250301931536, "checkpoint_time": "2025-04-04 15:41:20.705", "error": null }' + echo '{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114250301931536, "checkpoint_time": "2025-04-04 15:41:20.705", "error": null }' { "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 457114250301931536, "checkpoint_time": "2025-04-04 15:41:20.705", "error": null } ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114250301931536, '"checkpoint_time":' '"2025-04-04' '15:41:20.705",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 457114250301931536, '"checkpoint_time":' '"2025-04-04' '15:41:20.705",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test2 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test2 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test2 -s Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:41:21.715 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 459, b: 459, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:24, }"] [row2="{ zFzkK: IsNull, d: 15:48:24, a: 459, b: 459, c: ME09Ntx2r, }"] [2025/04/04 15:41:21.815 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 460, b: 460, c: 5FDmG4FB, zFzkK: IsNull, d: 15:48:25, }"] wait process cdc.test exit for 1-th time... + info='{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114250354360331, "checkpoint_time": "2025-04-04 15:41:20.905", "error": null }' + echo '{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114250354360331, "checkpoint_time": "2025-04-04 15:41:20.905", "error": null }' { "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 457114250354360331, "checkpoint_time": "2025-04-04 15:41:20.905", "error": null } ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114250354360331, '"checkpoint_time":' '"2025-04-04' '15:41:20.905",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 457114250354360331, '"checkpoint_time":' '"2025-04-04' '15:41:20.905",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test3 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test3 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test3 -s [2025/04/04 15:41:21.915 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 461, b: 461, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:26, }"] [row2="{ a: 461, b: 461, c: nj, zFzkK: IsNull, d: 15:48:26, }"] [2025/04/04 15:41:22.015 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:27, a: 462, b: 462, }"] [row2="{ b: 462, c: bOajaAEZn1, zFzkK: IsNull, d: 15:48:27, a: 462, }"] [2025/04/04 15:41:22.115 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:28, a: 463, b: 463, }"] [row2="{ a: 463, b: 463, c: 2BYuyniLe, zFzkK: IsNull, d: 15:48:28, }"] wait process cdc.test exit for 2-th time... + info='{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114250616504331, "checkpoint_time": "2025-04-04 15:41:21.905", "error": null }' + echo '{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114250616504331, "checkpoint_time": "2025-04-04 15:41:21.905", "error": null }' { "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 457114250616504331, "checkpoint_time": "2025-04-04 15:41:21.905", "error": null } ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114250616504331, '"checkpoint_time":' '"2025-04-04' '15:41:21.905",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 457114250616504331, '"checkpoint_time":' '"2025-04-04' '15:41:21.905",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test4 normal null /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test4 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test4 -s [2025/04/04 15:41:22.216 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:48:29, a: 464, b: 464, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: 3ZlgM, zFzkK: IsNull, d: 15:48:29, a: 464, b: 464, }"] [2025/04/04 15:41:22.316 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:30, a: 465, b: 465, }"] [row2="{ b: 465, c: 2yqb5SvU, zFzkK: IsNull, d: 15:48:30, a: 465, }"] + set +x table test.multi_cdc1 exists table test.multi_cdc2 exists [2025/04/04 15:41:22.416 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 466, b: 466, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:31, }"] [row2="{ a: 466, b: 466, c: Ml, zFzkK: IsNull, d: 15:48:31, }"] [2025/04/04 15:41:22.516 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:48:32, a: 467, b: 467, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ zFzkK: IsNull, d: 15:48:32, a: 467, b: 467, c: G0YFtwFKtJ, }"] [2025/04/04 15:41:22.616 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 468, b: 468, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:33, }"] [row2="{ b: 468, c: MAXOi, zFzkK: IsNull, d: 15:48:33, a: 468, }"] + info='{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114250616504331, "checkpoint_time": "2025-04-04 15:41:21.905", "error": null }' + echo '{ "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114250616504331, "checkpoint_time": "2025-04-04 15:41:21.905", "error": null }' { "upstream_id": 7489359808814298390, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 457114250616504331, "checkpoint_time": "2025-04-04 15:41:21.905", "error": null } ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114250616504331, '"checkpoint_time":' '"2025-04-04' '15:41:21.905",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359808814298390, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 457114250616504331, '"checkpoint_time":' '"2025-04-04' '15:41:21.905",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit sql_check run sql_check, 127.0.0.1 run task successfully run test case test_stop_processor [Fri Apr 4 15:41:22 CST 2025] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.64066408.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_stop_processor.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_stop_processor --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:22 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.23761.out cli changefeed create --start-ts=457114248635482113 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' table ddl_attributes.finish_mark not exists for 4-th check, retry later check diff failed 1-th time, retry later [2025/04/04 15:41:22.716 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:34, a: 469, b: 469, c: Nw9p, }"] [row2="{ zFzkK: IsNull, d: 15:48:34, a: 469, b: 469, c: aPx1eT, }"] [2025/04/04 15:41:22.817 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 470, b: 470, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:36, }"] [row2="{ a: 470, b: 470, c: srnTvLCq5, zFzkK: IsNull, d: 15:48:36, }"] Create changefeed successfully! ID: 267eab32-43d3-4f93-94aa-2e4de3887bf3 Info: {"upstream_id":7489359806631994089,"namespace":"default","id":"267eab32-43d3-4f93-94aa-2e4de3887bf3","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:41:22.917919188+08:00","start_ts":457114248635482113,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114248635482113,"checkpoint_ts":457114248635482113,"checkpoint_time":"2025-04-04 15:41:14.348"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:22.917 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 471, b: 471, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:37, }"] [row2="{ a: 471, b: 471, c: 7o0fo9, zFzkK: IsNull, d: 15:48:37, }"] [2025/04/04 15:41:23.017 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:48:38, a: 472, b: 472, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 472, b: 472, c: nGrijIdP, zFzkK: IsNull, d: 15:48:38, }"] [2025/04/04 15:41:23.117 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 473, b: 473, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:39, }"] [row2="{ zFzkK: IsNull, d: 15:48:39, a: 473, b: 473, c: m3REM, }"] changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11859.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:23.217 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 474, b: 474, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:40, }"] [row2="{ b: 474, c: xX, zFzkK: IsNull, d: 15:48:40, a: 474, }"] [2025/04/04 15:41:23.318 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 475, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:41, a: 475, }"] [row2="{ d: 15:48:41, a: 475, b: 475, c: W60, zFzkK: IsNull, }"] Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489359713796286682,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:23.27692578+08:00","start_ts":457114250948640773,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114250948640773,"checkpoint_ts":457114250948640773,"checkpoint_time":"2025-04-04 15:41:23.172"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:23.418 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 476, b: 476, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:42, }"] [row2="{ a: 476, b: 476, c: NCC, zFzkK: IsNull, d: 15:48:42, }"] [2025/04/04 15:41:23.518 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:48:43, a: 477, b: 477, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ b: 477, c: XA, zFzkK: IsNull, d: 15:48:43, a: 477, }"] [2025/04/04 15:41:23.618 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:48:44, a: 478, b: 478, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ b: 478, c: YwF7lhh, zFzkK: IsNull, d: 15:48:44, a: 478, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:23.718 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:45, a: 479, b: 479, }"] [row2="{ zFzkK: IsNull, d: 15:48:45, a: 479, b: 479, c: Ff, }"] [2025/04/04 15:41:23.818 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 480, b: 480, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:46, }"] [row2="{ zFzkK: IsNull, d: 15:48:46, a: 480, b: 480, c: qMP7mDMR, }"] [2025/04/04 15:41:23.919 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 481, b: 481, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:47, }"] [row2="{ b: 481, c: OJuqdW, zFzkK: IsNull, d: 15:48:47, a: 481, }"] Create changefeed successfully! ID: test-2 Info: {"upstream_id":7489359462573592486,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:41:23.71921016+08:00","start_ts":457114249178382337,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114249178382337,"checkpoint_ts":457114249178382337,"checkpoint_time":"2025-04-04 15:41:16.419"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:24.019 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 482, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:48, a: 482, }"] [row2="{ c: JAHm15bJJW, zFzkK: IsNull, d: 15:48:48, a: 482, b: 482, }"] [2025/04/04 15:41:24.119 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 483, b: 483, c: TIDIuPDq, zFzkK: IsNull, d: 15:48:49, }"] + set +x [Fri Apr 4 15:41:24 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2380023802.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8301 failed: Connection refused * Failed to connect to 127.0.0.1 port 8301: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:24.219 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:50, a: 484, b: 484, }"] [row2="{ a: 484, b: 484, c: a7yZ3, zFzkK: IsNull, d: 15:48:50, }"] [2025/04/04 15:41:24.319 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ d: 15:48:51, a: 485, b: 485, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 485, b: 485, c: SorSM4A, zFzkK: IsNull, d: 15:48:51, }"] [2025/04/04 15:41:24.420 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 486, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:52, a: 486, }"] [row2="{ b: 486, c: 2, zFzkK: IsNull, d: 15:48:52, a: 486, }"] [2025/04/04 15:41:24.520 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:53, a: 487, b: 487, c: Nw9p, }"] [row2="{ a: 487, b: 487, c: VloCI, zFzkK: IsNull, d: 15:48:53, }"] [2025/04/04 15:41:24.620 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:54, a: 488, b: 488, c: Nw9p, }"] [row2="{ a: 488, b: 488, c: C1rgKUF, zFzkK: IsNull, d: 15:48:54, }"] + set +x [2025/04/04 15:41:24.720 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 489, b: 489, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:55, }"] [row2="{ a: 489, b: 489, c: ofcmK, zFzkK: IsNull, d: 15:48:55, }"] [2025/04/04 15:41:24.820 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 490, b: 490, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:57, }"] [row2="{ a: 490, b: 490, c: w0clf8, zFzkK: IsNull, d: 15:48:57, }"] [2025/04/04 15:41:24.921 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 491, b: 491, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:58, }"] [row2="{ a: 491, b: 491, c: iXpS2, zFzkK: IsNull, d: 15:48:58, }"] [2025/04/04 15:41:25.021 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:48:59, a: 492, b: 492, }"] [row2="{ a: 492, b: 492, c: 8p, zFzkK: IsNull, d: 15:48:59, }"] [2025/04/04 15:41:25.121 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 493, b: 493, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:00, }"] [row2="{ a: 493, b: 493, c: mEYFNfk0M, zFzkK: IsNull, d: 15:49:00, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 5-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... + set +x + run_sql 'use test; create table t2(id bigint primary key, a date); insert into t2 values(1, '\''2023-02-08'\'');' 127.0.0.1 4000 [2025/04/04 15:41:25.221 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 494, b: 494, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:01, }"] [row2="{ b: 494, c: VBQsmA4a, zFzkK: IsNull, d: 15:49:01, a: 494, }"] [2025/04/04 15:41:25.321 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 495, b: 495, c: MKX, zFzkK: IsNull, d: 15:49:02, }"] [2025/04/04 15:41:25.421 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:03, a: 496, b: 496, }"] [row2="{ d: 15:49:03, a: 496, b: 496, c: 4R6SKS, zFzkK: IsNull, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:25.522 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 497, b: 497, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:04, }"] [row2="{ b: 497, c: gpfvovbnPB, zFzkK: IsNull, d: 15:49:04, a: 497, }"] [2025/04/04 15:41:25.622 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 498, b: 498, c: zLEA087, zFzkK: IsNull, d: 15:49:05, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:41:25.722 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 499, b: 499, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:06, }"] [row2="{ a: 499, b: 499, c: z9ogINXq, zFzkK: IsNull, d: 15:49:06, }"] [2025/04/04 15:41:25.822 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 500, b: 500, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:09, }"] [2025/04/04 15:41:25.922 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 501, b: 501, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:10, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004878630 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114251464802309 MinTableBarrierTs:457114251464802310 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114251464802310, checkpointTs: 457114251464802310, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114251464802309, checkpointTs: 457114251464802309, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114251464802310, checkpointTs: 457114251464802310, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6267a15-747b-48b2-ac81-15775ed23c6d {"id":"e6267a15-747b-48b2-ac81-15775ed23c6d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752483} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe6d4 e6267a15-747b-48b2-ac81-15775ed23c6d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114251464802309,"min-table-barrier-ts":457114251464802310,"admin-job-type":0} /tidb/cdc/default/default/task/position/e6267a15-747b-48b2-ac81-15775ed23c6d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004878630 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114251464802309 MinTableBarrierTs:457114251464802310 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114251464802310, checkpointTs: 457114251464802310, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114251464802309, checkpointTs: 457114251464802309, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114251464802310, checkpointTs: 457114251464802310, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6267a15-747b-48b2-ac81-15775ed23c6d {"id":"e6267a15-747b-48b2-ac81-15775ed23c6d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752483} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe6d4 e6267a15-747b-48b2-ac81-15775ed23c6d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114251464802309,"min-table-barrier-ts":457114251464802310,"admin-job-type":0} /tidb/cdc/default/default/task/position/e6267a15-747b-48b2-ac81-15775ed23c6d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {UpstreamID:7489358766378819224 Namespace:default ID:cb28ccbc-03e8-4199-beb7-a5edfeec8da7 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:37:19.080158588 +0800 CST StartTs:457114185234120705 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004878630 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114186951163912} {CheckpointTs:457114251464802309 MinTableBarrierTs:457114251464802310 AdminJobType:noop} span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114251464802310, checkpointTs: 457114251464802310, state: Replicating span: {tableID:112, startKey:7480000000000000ff705f720000000000fa, endKey:7480000000000000ff705f730000000000fa}, resolvedTs: 457114251464802309, checkpointTs: 457114251464802309, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114251464802310, checkpointTs: 457114251464802310, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6267a15-747b-48b2-ac81-15775ed23c6d {"id":"e6267a15-747b-48b2-ac81-15775ed23c6d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752483} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/223195ffbb4fe6d4 e6267a15-747b-48b2-ac81-15775ed23c6d /tidb/cdc/default/default/changefeed/info/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"upstream-id":7489358766378819224,"namespace":"default","changefeed-id":"cb28ccbc-03e8-4199-beb7-a5edfeec8da7","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:37:19.080158588+08:00","start-ts":457114185234120705,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114186951163912} /tidb/cdc/default/default/changefeed/status/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":457114251464802309,"min-table-barrier-ts":457114251464802310,"admin-job-type":0} /tidb/cdc/default/default/task/position/e6267a15-747b-48b2-ac81-15775ed23c6d/cb28ccbc-03e8-4199-beb7-a5edfeec8da7 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489358766378819224 {"id":7489358766378819224,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2025/04/04 15:41:26.022 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:11, a: 502, b: 502, c: Nw9p, }"] [2025/04/04 15:41:26.123 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:12, a: 503, b: 503, }"] [2025/04/04 15:41:26.223 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 504, b: 504, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:13, }"] [2025/04/04 15:41:26.323 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 505, b: 505, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:14, }"] [2025/04/04 15:41:26.423 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 506, b: 506, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:15, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:26 CST 2025] <<<<<< run test case multi_cdc_cluster success! >>>>>> [2025/04/04 15:41:26.523 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:16, a: 507, b: 507, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:26.623 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:17, a: 508, b: 508, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] table test.simple1 exists running list_changefeed pass test: list changefeed [2025/04/04 15:41:26.724 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 509, b: 509, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:18, }"] [2025/04/04 15:41:26.824 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 510, b: 510, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:19, }"] [2025/04/04 15:41:26.924 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:20, a: 511, b: 511, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 6-th check, retry later [2025/04/04 15:41:27.024 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:21, a: 512, b: 512, }"] [2025/04/04 15:41:27.124 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 513, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:22, a: 513, }"] "is-owner": true, run task successfully [2025/04/04 15:41:27.225 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:23, a: 514, b: 514, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:27.325 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:24, a: 515, b: 515, }"] [2025/04/04 15:41:27.425 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:25, a: 516, b: 516, c: Nw9p, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8301 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:27 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/267eab32-43d3-4f93-94aa-2e4de3887bf3 {UpstreamID:7489359806631994089 Namespace:default ID:267eab32-43d3-4f93-94aa-2e4de3887bf3 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:41:22.917919188 +0800 CST StartTs:457114248635482113 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003dd5830 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114250876551172} {CheckpointTs:457114249317318724 MinTableBarrierTs:457114251859591170 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/capture/f49e559b-b008-4451-9fd3-0bd465864a43 {"id":"f49e559b-b008-4451-9fd3-0bd465864a43","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6959 f49e559b-b008-4451-9fd3-0bd465864a43 /tidb/cdc/default/default/changefeed/info/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"upstream-id":7489359806631994089,"namespace":"default","changefeed-id":"267eab32-43d3-4f93-94aa-2e4de3887bf3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:41:22.917919188+08:00","start-ts":457114248635482113,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114250876551172} /tidb/cdc/default/default/changefeed/status/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":457114249317318724,"min-table-barrier-ts":457114251859591170,"admin-job-type":0} /tidb/cdc/default/default/task/position/0386c122-8e01-4b7e-bc6e-15406a3750e2/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f49e559b-b008-4451-9fd3-0bd465864a43/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/267eab32-43d3-4f93-94aa-2e4de3887bf3 {UpstreamID:7489359806631994089 Namespace:default ID:267eab32-43d3-4f93-94aa-2e4de3887bf3 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:41:22.917919188 +0800 CST StartTs:457114248635482113 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003dd5830 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114250876551172} {CheckpointTs:457114249317318724 MinTableBarrierTs:457114251859591170 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/capture/f49e559b-b008-4451-9fd3-0bd465864a43 {"id":"f49e559b-b008-4451-9fd3-0bd465864a43","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6959 f49e559b-b008-4451-9fd3-0bd465864a43 /tidb/cdc/default/default/changefeed/info/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"upstream-id":7489359806631994089,"namespace":"default","changefeed-id":"267eab32-43d3-4f93-94aa-2e4de3887bf3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:41:22.917919188+08:00","start-ts":457114248635482113,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114250876551172} /tidb/cdc/default/default/changefeed/status/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":457114249317318724,"min-table-barrier-ts":457114251859591170,"admin-job-type":0} /tidb/cdc/default/default/task/position/0386c122-8e01-4b7e-bc6e-15406a3750e2/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f49e559b-b008-4451-9fd3-0bd465864a43/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/267eab32-43d3-4f93-94aa-2e4de3887bf3 {UpstreamID:7489359806631994089 Namespace:default ID:267eab32-43d3-4f93-94aa-2e4de3887bf3 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:41:22.917919188 +0800 CST StartTs:457114248635482113 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003dd5830 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114250876551172} {CheckpointTs:457114249317318724 MinTableBarrierTs:457114251859591170 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/capture/f49e559b-b008-4451-9fd3-0bd465864a43 {"id":"f49e559b-b008-4451-9fd3-0bd465864a43","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6959 f49e559b-b008-4451-9fd3-0bd465864a43 /tidb/cdc/default/default/changefeed/info/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"upstream-id":7489359806631994089,"namespace":"default","changefeed-id":"267eab32-43d3-4f93-94aa-2e4de3887bf3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:41:22.917919188+08:00","start-ts":457114248635482113,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114250876551172} /tidb/cdc/default/default/changefeed/status/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":457114249317318724,"min-table-barrier-ts":457114251859591170,"admin-job-type":0} /tidb/cdc/default/default/task/position/0386c122-8e01-4b7e-bc6e-15406a3750e2/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f49e559b-b008-4451-9fd3-0bd465864a43/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 4 15:41:27 CST 2025] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2386523867.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8302 failed: Connection refused * Failed to connect to 127.0.0.1 port 8302: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:27.525 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 517, b: 517, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:26, }"] [2025/04/04 15:41:27.625 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:27, a: 518, b: 518, c: Nw9p, }"] running get_changefeed [2025/04/04 15:41:27.726 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 519, b: 519, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:28, }"] [2025/04/04 15:41:27.826 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:49:29, a: 520, b: 520, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 520, b: 520, c: myiqBglaKd, zFzkK: 1BDrCP, d: 15:49:29, }"] [2025/04/04 15:41:27.926 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 521, b: 521, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:30, }"] [row2="{ a: 521, b: 521, c: 4GScR3d, zFzkK: N07rnSXbmKrkdsyVfx87BupimAwy9R33xisyR6e, d: 15:49:30, }"] pass test: get changefeed + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114252075859974, "checkpoint_time": "2025-04-04 15:41:27.472", "error": null }' + echo '{ "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114252075859974, "checkpoint_time": "2025-04-04 15:41:27.472", "error": null }' { "upstream_id": 7489359713796286682, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114252075859974, "checkpoint_time": "2025-04-04 15:41:27.472", "error": null } ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114252075859974, '"checkpoint_time":' '"2025-04-04' '15:41:27.472",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489359713796286682, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114252075859974, '"checkpoint_time":' '"2025-04-04' '15:41:27.472",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359866362132953 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd6dec0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:10607, start at 2025-04-04 15:41:25.797620465 +0800 CST m=+2.272215317 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:25.845 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:25.836 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:25.836 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359866362132953 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd6dec0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:10607, start at 2025-04-04 15:41:25.797620465 +0800 CST m=+2.272215317 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:25.845 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:25.836 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:25.836 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359874771838772 TiDB Cluster ID. tikv_gc_leader_uuid 657fefd6f78000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-m04ht-46fh4-cv3zj, pid:10671, start at 2025-04-04 15:41:25.875154705 +0800 CST m=+2.283996315 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:25.883 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:25.854 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:25.854 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:41:28.026 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:31, a: 522, b: 522, c: Nw9p, }"] [row2="{ a: 522, b: 522, c: 8I5uyh, zFzkK: nCMxXCwZSOdV47OQ1EdgiA11R, d: 15:49:31, }"] [2025/04/04 15:41:28.126 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 523, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:32, a: 523, }"] [row2="{ a: 523, b: 523, c: G, zFzkK: f7r6HNTR5gmgJD1Ib7EItxUSW5CvGmUkd9HvjsoVt, d: 15:49:32, }"] [2025/04/04 15:41:28.227 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:33, a: 524, b: 524, }"] [row2="{ zFzkK: DwezDblO0kGvkGyUC0NM, d: 15:49:33, a: 524, b: 524, c: qRkfWP, }"] [2025/04/04 15:41:28.327 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:34, a: 525, b: 525, }"] [row2="{ c: OI91YFOI3, zFzkK: BsaajvwoT5uYvGjAAOvWdUVe7SiByvmDehvJtxJMUX, d: 15:49:34, a: 525, b: 525, }"] [2025/04/04 15:41:28.427 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 526, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:35, a: 526, }"] [row2="{ a: 526, b: 526, c: M8c, zFzkK: tfZT, d: 15:49:35, }"] [2025/04/04 15:41:28.527 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 527, b: 527, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:36, }"] [row2="{ d: 15:49:36, a: 527, b: 527, c: IWD, zFzkK: O1Ul6Cx7OhMP8O5tA7K6uteHAdYHZeajDfpMhnXxjK, }"] [2025/04/04 15:41:28.627 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 528, b: 528, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:37, }"] [row2="{ a: 528, b: 528, c: s4ObxXUy, zFzkK: pE, d: 15:49:37, }"] [2025/04/04 15:41:28.727 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 529, b: 529, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:38, }"] [row2="{ a: 529, b: 529, c: R0ZRkix, zFzkK: tgaX4PcwLqw8o5VzpmAutw7IiRSk9, d: 15:49:38, }"] [2025/04/04 15:41:28.828 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 530, b: 530, c: WuRi4Ppqbv, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:39, }"] [2025/04/04 15:41:28.928 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: MNfv5FXiMh, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:40, a: 531, b: 531, }"] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11960.out cli unsafe delete-service-gc-safepoint table ddl_attributes.finish_mark not exists for 7-th check, retry later [2025/04/04 15:41:29.028 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: YIW6g2aVN, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:41, a: 532, b: 532, }"] [2025/04/04 15:41:29.128 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:42, a: 533, b: 533, c: j, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:29.228 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 534, b: 534, c: 6W, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:43, }"] owner pid: 6411 owner id e6267a15-747b-48b2-ac81-15775ed23c6d Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... running pause_changefeed [2025/04/04 15:41:29.329 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:44, a: 535, b: 535, c: 5fISmPISD, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:29.429 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:45, a: 536, b: 536, c: tx, }"] pass test: pause changefeed ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:41:29.529 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 537, b: 537, c: ZfohkoSUBJ, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:46, }"] [2025/04/04 15:41:29.630 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 538, b: 538, c: i8W, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:47, }"] [2025/04/04 15:41:29.730 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:48, a: 539, b: 539, c: BIEDj6Aszp, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:29.830 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 540, b: 540, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:49, }"] [2025/04/04 15:41:29.930 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 541, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:50, a: 541, }"] [Fri Apr 4 15:41:29 CST 2025] <<<<<< START cdc server in owner_remove_table_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/OwnerRemoveTableError=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_remove_table_error.1191411916.out server --log-file /tmp/tidb_cdc_test/owner_remove_table_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_remove_table_error/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:30.031 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:51, a: 542, b: 542, }"] [2025/04/04 15:41:30.131 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:52, a: 543, b: 543, c: Nw9p, }"] [2025/04/04 15:41:30.231 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 544, b: 544, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:53, }"] [2025/04/04 15:41:30.331 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 545, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:54, a: 545, }"] [2025/04/04 15:41:30.431 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:55, a: 546, b: 546, c: Nw9p, }"] running update_changefeed changefeed id: cb28ccbc-03e8-4199-beb7-a5edfeec8da7 curl -s -d "cf-id=cb28ccbc-03e8-4199-beb7-a5edfeec8da7&admin-job=1" http://127.0.0.1:8300/capture/owner/admin | grep true "status": true, run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 89 100 35 100 54 500 771 --:--:-- --:--:-- --:--:-- 1271 { "status": true, "message": "" }nonempty select id, val from test.availability1 where id=4 and val=4 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.11992.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 [2025/04/04 15:41:30.531 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 547, b: 547, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:56, }"] [2025/04/04 15:41:30.632 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:57, a: 548, b: 548, }"] [2025/04/04 15:41:30.732 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:49:58, a: 549, b: 549, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] pass test: update changefeed reset and all metadata truncated in PD! PASS + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8302 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/267eab32-43d3-4f93-94aa-2e4de3887bf3 {UpstreamID:7489359806631994089 Namespace:default ID:267eab32-43d3-4f93-94aa-2e4de3887bf3 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:41:22.917919188 +0800 CST StartTs:457114248635482113 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e0cea0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114250876551172} {CheckpointTs:457114249500295233 MinTableBarrierTs:457114252646023174 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/capture/9cd74e7f-52d8-447d-85c9-9d8eef346d73 {"id":"9cd74e7f-52d8-447d-85c9-9d8eef346d73","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752489} /tidb/cdc/default/__cdc_meta__/capture/f49e559b-b008-4451-9fd3-0bd465864a43 {"id":"f49e559b-b008-4451-9fd3-0bd465864a43","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6959 f49e559b-b008-4451-9fd3-0bd465864a43 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc69a2 9cd74e7f-52d8-447d-85c9-9d8eef346d73 /tidb/cdc/default/default/changefeed/info/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"upstream-id":7489359806631994089,"namespace":"default","changefeed-id":"267eab32-43d3-4f93-94aa-2e4de3887bf3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:41:22.917919188+08:00","start-ts":457114248635482113,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114250876551172} /tidb/cdc/default/default/changefeed/status/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":457114249500295233,"min-table-barrier-ts":457114252646023174,"admin-job-type":0} /tidb/cdc/default/default/task/position/0386c122-8e01-4b7e-bc6e-15406a3750e2/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9cd74e7f-52d8-447d-85c9-9d8eef346d73/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f49e559b-b008-4451-9fd3-0bd465864a43/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/267eab32-43d3-4f93-94aa-2e4de3887bf3 {UpstreamID:7489359806631994089 Namespace:default ID:267eab32-43d3-4f93-94aa-2e4de3887bf3 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:41:22.917919188 +0800 CST StartTs:457114248635482113 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e0cea0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114250876551172} {CheckpointTs:457114249500295233 MinTableBarrierTs:457114252646023174 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/capture/9cd74e7f-52d8-447d-85c9-9d8eef346d73 {"id":"9cd74e7f-52d8-447d-85c9-9d8eef346d73","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752489} /tidb/cdc/default/__cdc_meta__/capture/f49e559b-b008-4451-9fd3-0bd465864a43 {"id":"f49e559b-b008-4451-9fd3-0bd465864a43","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6959 f49e559b-b008-4451-9fd3-0bd465864a43 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc69a2 9cd74e7f-52d8-447d-85c9-9d8eef346d73 /tidb/cdc/default/default/changefeed/info/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"upstream-id":7489359806631994089,"namespace":"default","changefeed-id":"267eab32-43d3-4f93-94aa-2e4de3887bf3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:41:22.917919188+08:00","start-ts":457114248635482113,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114250876551172} /tidb/cdc/default/default/changefeed/status/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":457114249500295233,"min-table-barrier-ts":457114252646023174,"admin-job-type":0} /tidb/cdc/default/default/task/position/0386c122-8e01-4b7e-bc6e-15406a3750e2/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9cd74e7f-52d8-447d-85c9-9d8eef346d73/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f49e559b-b008-4451-9fd3-0bd465864a43/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359806631994089 {"id":7489359806631994089,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/267eab32-43d3-4f93-94aa-2e4de3887bf3 {UpstreamID:7489359806631994089 Namespace:default ID:267eab32-43d3-4f93-94aa-2e4de3887bf3 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2025-04-04 15:41:22.917919188 +0800 CST StartTs:457114248635482113 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003e0cea0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114250876551172} {CheckpointTs:457114249500295233 MinTableBarrierTs:457114252646023174 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0386c122-8e01-4b7e-bc6e-15406a3750e2 {"id":"0386c122-8e01-4b7e-bc6e-15406a3750e2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752479} /tidb/cdc/default/__cdc_meta__/capture/9cd74e7f-52d8-447d-85c9-9d8eef346d73 {"id":"9cd74e7f-52d8-447d-85c9-9d8eef346d73","address":"127.0.0.1:8302","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752489} /tidb/cdc/default/__cdc_meta__/capture/f49e559b-b008-4451-9fd3-0bd465864a43 {"id":"f49e559b-b008-4451-9fd3-0bd465864a43","address":"127.0.0.1:8301","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752484} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6907 0386c122-8e01-4b7e-bc6e-15406a3750e2 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc6959 f49e559b-b008-4451-9fd3-0bd465864a43 /tidb/cdc/default/__cdc_meta__/owner/223195ffbefc69a2 9cd74e7f-52d8-447d-85c9-9d8eef346d73 /tidb/cdc/default/default/changefeed/info/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"upstream-id":7489359806631994089,"namespace":"default","changefeed-id":"267eab32-43d3-4f93-94aa-2e4de3887bf3","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2025-04-04T15:41:22.917919188+08:00","start-ts":457114248635482113,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114250876551172} /tidb/cdc/default/default/changefeed/status/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":457114249500295233,"min-table-barrier-ts":457114252646023174,"admin-job-type":0} /tidb/cdc/default/default/task/position/0386c122-8e01-4b7e-bc6e-15406a3750e2/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9cd74e7f-52d8-447d-85c9-9d8eef346d73/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/f49e559b-b008-4451-9fd3-0bd465864a43/267eab32-43d3-4f93-94aa-2e4de3887bf3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489359806631994089 + grep -q 'etcd info' {"id":7489359806631994089,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table ddl_attributes.finish_mark not exists for 8-th check, retry later [2025/04/04 15:41:30.832 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:49:59, a: 550, b: 550, c: Nw9p, }"] [2025/04/04 15:41:30.932 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 551, b: 551, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:00, }"] coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:31.032 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 552, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:01, a: 552, }"] [2025/04/04 15:41:31.133 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 553, b: 553, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:02, }"] [2025/04/04 15:41:31.233 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 554, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:03, a: 554, }"] table move_table.usertable exists [2025/04/04 15:41:31.333 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:50:04, a: 555, b: 555, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:31.433 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 556, b: 556, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:05, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359801972423932 TiDB Cluster ID. tikv_gc_leader_uuid 657fefda640000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:9605, start at 2025-04-04 15:41:29.375399454 +0800 CST m=+22.494141080 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:29.383 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:29.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:29.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359801972423932 TiDB Cluster ID. tikv_gc_leader_uuid 657fefda640000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:9605, start at 2025-04-04 15:41:29.375399454 +0800 CST m=+22.494141080 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:29.383 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:29.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:29.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489359803514842230 TiDB Cluster ID. tikv_gc_leader_uuid 657fefc82c0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:9671, start at 2025-04-04 15:41:10.751310139 +0800 CST m=+3.793053538 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:43:10.759 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:41:10.754 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:31:10.754 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/benbjohnson/clock v1.1.0 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/go-mysql-org/go-mysql v1.12.0 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading gorm.io/gorm v1.25.11 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 [2025/04/04 15:41:31.533 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 557, b: 557, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:06, }"] [2025/04/04 15:41:31.633 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 558, b: 558, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:07, }"] [2025/04/04 15:41:31.733 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:08, a: 559, b: 559, }"] go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/gin-gonic/gin v1.9.1 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/pelletier/go-toml/v2 v2.0.8 go: downloading github.com/ugorji/go/codec v1.2.11 go: downloading github.com/go-playground/validator/v10 v10.14.0 [2025/04/04 15:41:31.834 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 560, b: 560, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:09, }"] [2025/04/04 15:41:31.934 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 562, b: 562, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:11, }"] running resume_changefeed go: downloading filippo.io/edwards25519 v1.1.0 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 [2025/04/04 15:41:32.034 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 563, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:12, a: 563, }"] [2025/04/04 15:41:32.134 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 564, b: 564, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:13, }"] [2025/04/04 15:41:32.234 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:14, a: 565, b: 565, c: Nw9p, }"] pass test: resume changefeed go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading golang.org/x/mod v0.22.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/leodido/go-urn v1.2.4 go: downloading github.com/gabriel-vasile/mimetype v1.4.2 go: downloading github.com/go-playground/universal-translator v0.18.1 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/go-playground/locales v0.14.1 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading go.opentelemetry.io/otel/sdk v1.24.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading go.etcd.io/bbolt v1.3.9 go: downloading github.com/golang-jwt/jwt/v4 v4.5.1 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/gorilla/websocket v1.5.1 nonempty select id, val from test.availability1 where id=4 and val=4 + set +x [2025/04/04 15:41:32.335 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 566, b: 566, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:15, }"] [2025/04/04 15:41:32.435 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 567, b: 567, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:16, }"] [2025/04/04 15:41:32.535 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:50:17, a: 568, b: 568, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 run task successfully test_stop_processor pass [2025/04/04 15:41:32.635 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 569, b: 569, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:18, }"] [2025/04/04 15:41:32.735 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 570, b: 570, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:19, }"] [row2="{ d: 15:50:19, a: 570, b: 570, c: RhIP, zFzkK: dWMWKrj6XNuGfSBD98mB8u1ouuV8Y, }"] table ddl_attributes.finish_mark not exists for 9-th check, retry later [2025/04/04 15:41:32.836 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 571, b: 571, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:20, }"] [row2="{ a: 571, b: 571, c: y, zFzkK: L4oJ6eLq2NtZRGUU72WK, d: 15:50:20, }"] [2025/04/04 15:41:32.936 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 572, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:21, a: 572, }"] [row2="{ c: yagW, zFzkK: BeO7hV9MqSgpT8AKjwpkAZP0A7xVFs04nii4qMR2jn1x, d: 15:50:21, a: 572, b: 572, }"] [2025/04/04 15:41:33.036 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:22, a: 573, b: 573, }"] [row2="{ c: l5XnFUs, zFzkK: HUTz4ZBGZcAy2Swv8g2kTXpE09VcwmXzv, d: 15:50:22, a: 573, b: 573, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:33 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fb1c0103-6174-4155-b858-6ceac67a9c90 {"id":"fb1c0103-6174-4155-b858-6ceac67a9c90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752490} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf3c97d9 fb1c0103-6174-4155-b858-6ceac67a9c90 /tidb/cdc/default/default/upstream/7489359866362132953 {"id":7489359866362132953,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fb1c0103-6174-4155-b858-6ceac67a9c90 {"id":"fb1c0103-6174-4155-b858-6ceac67a9c90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752490} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf3c97d9 fb1c0103-6174-4155-b858-6ceac67a9c90 /tidb/cdc/default/default/upstream/7489359866362132953 {"id":7489359866362132953,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fb1c0103-6174-4155-b858-6ceac67a9c90 {"id":"fb1c0103-6174-4155-b858-6ceac67a9c90","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752490} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbf3c97d9 fb1c0103-6174-4155-b858-6ceac67a9c90 /tidb/cdc/default/default/upstream/7489359866362132953 {"id":7489359866362132953,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:41:33.136 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 574, b: 574, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:23, }"] [row2="{ a: 574, b: 574, c: MVII, zFzkK: e9RC6d1AIzpQiVzcxK5YCImAm5wdStbqaR96iqko, d: 15:50:23, }"] [2025/04/04 15:41:33.236 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 575, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:24, a: 575, }"] [row2="{ a: 575, b: 575, c: ksmVVetq, zFzkK: NCyqE5VeV8CWC2jeGUzhCoPrHzM14UNW6yEafVynH, d: 15:50:24, }"] wait process cdc.test exit for 1-th time... running rebalance_table [Fri Apr 4 15:41:33 CST 2025] <<<<<< START cdc server in multi_rocks case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.2347023472.out server --log-file /tmp/tidb_cdc_test/multi_rocks/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_rocks/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:41:33.336 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 576, b: 576, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:25, }"] [row2="{ a: 576, b: 576, c: 1fLK, zFzkK: kUYDVfg6QgDuLzrU7uKB6mtqWMVqRrV4nFrjS, d: 15:50:25, }"] [2025/04/04 15:41:33.437 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 577, b: 577, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:26, }"] [row2="{ a: 577, b: 577, c: nbnQJ, zFzkK: ECLgMVceLEBE5FJoyWbqeb30Ae3DWi, d: 15:50:26, }"] [2025/04/04 15:41:33.537 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 578, b: 578, c: 2f, zFzkK: bLvRYZE8KxN6IuexTBcgHLpsi9tt0y, d: 15:50:27, }"] wait process cdc.test exit for 2-th time... pass test: rebalance table [2025/04/04 15:41:33.637 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 579, b: 579, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:28, }"] [row2="{ a: 579, b: 579, c: wmd, zFzkK: yDU9S59AeA8r6ZT4, d: 15:50:28, }"] [2025/04/04 15:41:33.737 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:50:29, a: 580, b: 580, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: zwRA67ASj, zFzkK: deXMoWtwA9iyynz335tXJXihFyeHEnxmswgp10kEd0qZ, d: 15:50:29, a: 580, b: 580, }"] [2025/04/04 15:41:33.837 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 581, b: 581, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:30, }"] [row2="{ a: 581, b: 581, c: rgNR, zFzkK: 2iPezgxES90, d: 15:50:30, }"] [2025/04/04 15:41:33.938 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 582, b: 582, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:31, }"] [row2="{ a: 582, b: 582, c: nb8, zFzkK: eUeFZBPihPXV62kBDHBVcPk2TOXR8OfH3NDc4eaM, d: 15:50:31, }"] [2025/04/04 15:41:34.038 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 583, b: 583, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:32, }"] [row2="{ a: 583, b: 583, c: SOhsXAXx, zFzkK: pv0Tg1m1WwDXnnhHVVVzjJoIv9iNpe9X4wbjlGWZFbDB, d: 15:50:32, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:34 CST 2025] <<<<<< run test case availability success! >>>>>> [2025/04/04 15:41:34.138 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 584, b: 584, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:33, }"] [row2="{ a: 584, b: 584, c: O3M7TlGv, zFzkK: 2FZ89nPBbR33Ukt9txwytt, d: 15:50:33, }"] [2025/04/04 15:41:34.238 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 585, b: 585, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:34, }"] [row2="{ c: Ktl9omvHi, zFzkK: 3aZGO3qqCdhZzGWWkarTmEVeXixbW1F3wJ1fA7yxbVpR, d: 15:50:34, a: 585, b: 585, }"] [2025/04/04 15:41:34.338 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 586, b: 586, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:35, }"] [row2="{ a: 586, b: 586, c: 0NPYU, zFzkK: RTDTPNUPbWLicYDCx3xaDDrdiYIGrB5IeSiqZ0, d: 15:50:35, }"] [2025/04/04 15:41:34.439 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 587, b: 587, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:36, }"] [row2="{ a: 587, b: 587, c: c, zFzkK: dXy6EeEw5iL9Yp3lSx, d: 15:50:36, }"] [2025/04/04 15:41:34.539 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 588, b: 588, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:37, }"] [row2="{ a: 588, b: 588, c: qDy, zFzkK: ZTOP0nmxQCB, d: 15:50:37, }"] [2025/04/04 15:41:34.639 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:38, a: 589, b: 589, }"] [row2="{ c: 4hc, zFzkK: TpBJZE6, d: 15:50:38, a: 589, b: 589, }"] [2025/04/04 15:41:34.739 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 590, b: 590, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:40, }"] [row2="{ zFzkK: aaYPBOh6Z9G0MGsxQ, d: 15:50:40, a: 590, b: 590, c: ZeLdo5Y, }"] running list_processor pass test: list processors [2025/04/04 15:41:34.839 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 591, b: 591, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:41, }"] [row2="{ b: 591, c: Zv, zFzkK: mPhfdi, d: 15:50:41, a: 591, }"] [2025/04/04 15:41:34.940 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:42, a: 592, b: 592, c: Nw9p, }"] [row2="{ a: 592, b: 592, c: XImNMXkUGY, zFzkK: 6, d: 15:50:42, }"] [2025/04/04 15:41:35.040 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:50:43, a: 593, b: 593, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 593, b: 593, c: G, zFzkK: orw7M58dM9, d: 15:50:43, }"] go: downloading github.com/ardielle/ardielle-go v1.5.2 table ddl_attributes.finish_mark exists check diff successfully [2025/04/04 15:41:35.140 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:44, a: 594, b: 594, }"] [row2="{ b: 594, c: aCMkYHWAfe, zFzkK: 0iTz, d: 15:50:44, a: 594, }"] [2025/04/04 15:41:35.240 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 595, b: 595, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:45, }"] [row2="{ zFzkK: B5NnhMx5gdcRSSjvtWXS8suW6XzG5NN7fDb0WDU7eDizI, d: 15:50:45, a: 595, b: 595, c: GsRvo, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:41:35.340 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 596, b: 596, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:46, }"] [row2="{ c: wo, zFzkK: k, d: 15:50:46, a: 596, b: 596, }"] [2025/04/04 15:41:35.440 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:47, a: 597, b: 597, }"] [row2="{ a: 597, b: 597, c: TaMtKy7vq, zFzkK: k8N354iseiY, d: 15:50:47, }"] [2025/04/04 15:41:35.541 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 598, b: 598, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:50:48, }"] [row2="{ a: 598, b: 598, c: nZh, zFzkK: 8mooKaWsNRdI68SkaCcSnn1PzoK4zYQmBNw9EkE, d: 15:50:48, }"] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12096.out cli unsafe resolve-lock --region=26 PASS [2025/04/04 15:41:35.641 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:50:49, a: 599, b: 599, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: wXt8fPBvfa, zFzkK: FHODIN91Bhvy4VD7z5dDyzVw, d: 15:50:49, a: 599, b: 599, }"] [2025/04/04 15:41:35.741 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 620, b: 620, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:14, }"] [row2="{ b: 620, c: E7SfZS, zFzkK: 0JFU, d: 15:51:14, a: 620, }"] running get_processor coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:41:35.841 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:15, a: 621, b: 621, }"] [row2="{ a: 621, b: 621, c: hzPk6ccjQq, zFzkK: 5v8YS8OmDXSArfS, d: 15:51:15, }"] [2025/04/04 15:41:35.941 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ b: 622, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:16, a: 622, }"] [row2="{ b: 622, c: zx53nK, zFzkK: Oj5aImSOMg6z5HtrNeyQOVcCUpnDK3aSRbkeihB5OL, d: 15:51:16, a: 622, }"] [2025/04/04 15:41:36.042 +08:00] [WARN] [diff.go:937] ["find different row"] [column=zFzkK] [row1="{ a: 623, b: 623, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:17, }"] [row2="{ zFzkK: GzqPd41In3P, d: 15:51:17, a: 623, b: 623, c: sXQXTeSV, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:41:36.142 +08:00] [WARN] [diff.go:940] ["target had superfluous data"] [row="{ a: 624, b: 624, c: NX15K, zFzkK: lpULJixUtku7WBUhcXuYsdSb35tdFSuX8e, d: 15:51:18, }"] [2025/04/04 15:41:36.242 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 625, b: 625, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:19, }"] [row2="{ zFzkK: HtnIxsz2S1mdO, d: 15:51:19, a: 625, b: 625, c: kwJDNLlgOz, }"] [2025/04/04 15:41:36.344 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:51:20, a: 626, b: 626, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: n4JDUbbN, zFzkK: ID6sD6XRyOjXNtmwybWOHsXvOKSFpkQWuCPPS4UKbF9, d: 15:51:20, a: 626, b: 626, }"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache table owner_remove_table_error.finished_mark not exists for 1-th check, retry later [Pipeline] } [2025/04/04 15:41:36.444 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 627, b: 627, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:21, }"] [row2="{ d: 15:51:21, a: 627, b: 627, c: YXy74, zFzkK: EMNGf01ASPOooaRysKtWPMLdrZfANshvmtpEdD, }"] [2025/04/04 15:41:36.544 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 628, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:22, a: 628, }"] [row2="{ c: spRDyZ45xX, zFzkK: x, d: 15:51:22, a: 628, b: 628, }"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:41:36 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd19d219-2664-44b1-8434-e341757f4a28 {"id":"dd19d219-2664-44b1-8434-e341757f4a28","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752493} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbef9d211 dd19d219-2664-44b1-8434-e341757f4a28 /tidb/cdc/default/default/upstream/7489359801972423932 {"id":7489359801972423932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd19d219-2664-44b1-8434-e341757f4a28 {"id":"dd19d219-2664-44b1-8434-e341757f4a28","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752493} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbef9d211 dd19d219-2664-44b1-8434-e341757f4a28 /tidb/cdc/default/default/upstream/7489359801972423932 {"id":7489359801972423932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dd19d219-2664-44b1-8434-e341757f4a28 {"id":"dd19d219-2664-44b1-8434-e341757f4a28","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752493} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbef9d211 dd19d219-2664-44b1-8434-e341757f4a28 /tidb/cdc/default/default/upstream/7489359801972423932 {"id":7489359801972423932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.cli.24737.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [Pipeline] // dir wait process cdc.test exit for 3-th time... [Pipeline] } [2025/04/04 15:41:36.644 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 629, b: 629, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:23, }"] [row2="{ zFzkK: APDjfK, d: 15:51:23, a: 629, b: 629, c: nUml, }"] [2025/04/04 15:41:36.744 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: 32zn1vLnumxmo, d: 15:51:24, a: 630, b: 630, c: u, }"] [2025/04/04 15:41:36.844 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 631, b: 631, c: qH, zFzkK: AmHg6NS5O5NYBDVmbn, d: 15:51:25, }"] [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [2025/04/04 15:41:36.945 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 632, b: 632, c: G, zFzkK: 0LI0eERC8sAB, d: 15:51:26, }"] [2025/04/04 15:41:37.045 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 633, c: 3oT, zFzkK: gNgGx2vj, d: 15:51:27, a: 633, }"] [Pipeline] // stage [Pipeline] } Create changefeed successfully! ID: a1070679-3067-4c62-873a-3a044393cb6e Info: {"upstream_id":7489359801972423932,"namespace":"default","id":"a1070679-3067-4c62-873a-3a044393cb6e","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:41:36.895005592+08:00","start_ts":457114254510915589,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114254510915589,"checkpoint_ts":457114254510915589,"checkpoint_time":"2025-04-04 15:41:36.761"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Pipeline] // container [2025/04/04 15:41:37.145 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 634, b: 634, c: HJ2, zFzkK: M82hZAbSIXnt9b5YeNGBXxUl, d: 15:51:28, }"] [2025/04/04 15:41:37.245 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 635, b: 635, c: zYAmsUV, zFzkK: 1m1WwAS6iYjE2jsKMc2K, d: 15:51:29, }"] [2025/04/04 15:41:37.346 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: a, d: 15:51:30, a: 636, b: 636, c: jlRK3fZ, }"] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_with_auth.cli.12130.out cli unsafe resolve-lock --region=26 --ts=457114253347520517 [Pipeline] } [Fri Apr 4 15:41:37 CST 2025] <<<<<< run test case ddl_attributes success! >>>>>> [Pipeline] // withEnv [Pipeline] } [2025/04/04 15:41:37.446 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 637, b: 637, c: RgHQz, zFzkK: WR466H, d: 15:51:31, }"] [2025/04/04 15:41:37.546 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 638, c: IhUb0Yo, zFzkK: ar5v8A0XRNHMBaD48Qrondt2hebP3O9, d: 15:51:32, a: 638, }"] PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [2025/04/04 15:41:37.646 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: erh3jmt, zFzkK: VPcLewQV7jvdmlKqVqRF8DxU8l9uq6HyT, d: 15:51:33, a: 639, b: 639, }"] [2025/04/04 15:41:37.746 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 640, b: 640, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:34, }"] [2025/04/04 15:41:37.847 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:35, a: 641, b: 641, }"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [2025/04/04 15:41:37.947 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:51:36, a: 642, b: 642, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:38.047 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:37, a: 643, b: 643, c: Nw9p, }"] [2025/04/04 15:41:38.147 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 644, b: 644, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:38, }"] [2025/04/04 15:41:38.247 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 645, b: 645, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:39, }"] [2025/04/04 15:41:38.348 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:40, a: 646, b: 646, }"] table owner_remove_table_error.finished_mark not exists for 2-th check, retry later pass test: get processor + set +x [2025/04/04 15:41:38.448 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:41, a: 647, b: 647, }"] [2025/04/04 15:41:38.548 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 648, b: 648, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:42, }"] [2025/04/04 15:41:38.648 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 649, b: 649, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:43, }"] [2025/04/04 15:41:38.748 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 650, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:44, a: 650, }"] [2025/04/04 15:41:38.848 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 651, b: 651, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:45, }"] + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 207k 6000 --:--:-- --:--:-- --:--:-- 212k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" check_redo_resolved_ts 7fe74488-ff74-4a3f-8d0c-cd20de57ea8e 457114254771224585 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e/meta global resolved ts 457114249292414984 not forward to 457114254771224585 run task failed 1-th time, retry later [2025/04/04 15:41:38.949 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 652, b: 652, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:46, }"] [2025/04/04 15:41:39.049 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 653, b: 653, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:47, }"] running move_table [2025/04/04 15:41:39.149 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 654, b: 654, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:48, }"] [2025/04/04 15:41:39.249 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 655, b: 655, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:49, }"] [2025/04/04 15:41:39.349 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 656, b: 656, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:50, }"] ***************** properties ***************** "threadcount"="2" "mysql.port"="4000" "workload"="core" "scanproportion"="0" "mysql.db"="multi_rocks" "mysql.user"="root" "insertproportion"="0" "updateproportion"="0" "operationcount"="0" "requestdistribution"="uniform" "recordcount"="1000" "dotransactions"="false" "readallfields"="true" "readproportion"="0" "table"="a1" "mysql.host"="127.0.0.1" ********************************************** pass test: move table [2025/04/04 15:41:39.450 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:51, a: 657, b: 657, c: Nw9p, }"] [2025/04/04 15:41:39.550 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 658, b: 658, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:52, }"] [2025/04/04 15:41:39.650 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 659, b: 659, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:53, }"] [2025/04/04 15:41:39.750 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:54, a: 660, b: 660, c: Nw9p, }"] [2025/04/04 15:41:39.851 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 661, b: 661, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:55, }"] check_redo_resolved_ts 7fe74488-ff74-4a3f-8d0c-cd20de57ea8e 457114254771224585 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e/meta [2025/04/04 15:41:39.951 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 662, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:56, a: 662, }"] [2025/04/04 15:41:40.051 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 663, b: 663, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:57, }"] global resolved ts 457114249292414984 not forward to 457114254771224585 run task failed 2-th time, retry later table owner_remove_table_error.finished_mark exists check diff successfully [2025/04/04 15:41:40.151 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:51:58, a: 664, b: 664, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:40.251 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:51:59, a: 665, b: 665, }"] [2025/04/04 15:41:40.352 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 666, b: 666, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:00, }"] Run finished, takes 699.587312ms INSERT - Takes(s): 0.7, Count: 1000, OPS: 1461.3, Avg(us): 1366, Min(us): 837, Max(us): 33235, 95th(us): 2000, 99th(us): 3000 running set_log_level [2025/04/04 15:41:40.452 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 668, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:02, a: 668, }"] [2025/04/04 15:41:40.552 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:03, a: 669, b: 669, }"] [2025/04/04 15:41:40.652 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:04, a: 670, b: 670, }"] pass test: set log level wait process cdc.test exit for 1-th time... [2025/04/04 15:41:40.752 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 671, b: 671, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:05, }"] [2025/04/04 15:41:40.853 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 672, b: 672, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:06, }"] ***************** properties ***************** "threadcount"="2" "readallfields"="true" "dotransactions"="false" "table"="a2" "insertproportion"="0" "readproportion"="0" "updateproportion"="0" "recordcount"="1000" "mysql.port"="4000" "requestdistribution"="uniform" "mysql.user"="root" "mysql.db"="multi_rocks" "operationcount"="0" "mysql.host"="127.0.0.1" "workload"="core" "scanproportion"="0" ********************************************** [2025/04/04 15:41:40.953 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 673, b: 673, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:07, }"] [2025/04/04 15:41:41.053 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 674, b: 674, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:08, }"] [2025/04/04 15:41:41.153 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 675, b: 675, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:09, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:41:41.253 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:10, a: 676, b: 676, }"] [2025/04/04 15:41:41.354 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 677, b: 677, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:11, }"] Run finished, takes 758.086918ms INSERT - Takes(s): 0.7, Count: 1000, OPS: 1352.6, Avg(us): 1481, Min(us): 946, Max(us): 39063, 95th(us): 2000, 99th(us): 3000 [2025/04/04 15:41:41.454 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 678, b: 678, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:12, }"] [2025/04/04 15:41:41.554 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 679, b: 679, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:13, }"] [2025/04/04 15:41:41.654 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 680, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:14, a: 680, }"] running remove_changefeed [2025/04/04 15:41:41.754 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 681, b: 681, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:15, }"] [2025/04/04 15:41:41.855 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 682, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:16, a: 682, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:41 CST 2025] <<<<<< run test case owner_remove_table_error success! >>>>>> ***************** properties ***************** "scanproportion"="0" "recordcount"="1000" "mysql.host"="127.0.0.1" "table"="a3" "readproportion"="0" "threadcount"="2" "updateproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "insertproportion"="0" "mysql.db"="multi_rocks" "mysql.port"="4000" "dotransactions"="false" "operationcount"="0" "workload"="core" "readallfields"="true" ********************************************** [2025/04/04 15:41:41.955 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:56:39, a: 683, b: 683, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:42.055 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 684, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:18, a: 684, }"] [2025/04/04 15:41:42.155 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 685, b: 685, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:19, }"] } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 178k 0 --:--:-- --:--:-- --:--:-- 178k { "version": "v9.0.0-beta.1-10-ga306f4894", "git_hash": "a306f48949e037965a7c1963f500f7e0abbfba51", "id": "ba24acc3-1010-41e3-ba78-41604517bec4", "pid": 11262, "is_owner": true pass test: remove changefeed [2025/04/04 15:41:42.255 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:20, a: 686, b: 686, c: Nw9p, }"] [2025/04/04 15:41:42.355 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 687, b: 687, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:57:42, }"] }wait process cdc.test exit for 1-th time... Run finished, takes 619.889082ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1661.2, Avg(us): 1207, Min(us): 826, Max(us): 17867, 95th(us): 2000, 99th(us): 2000 [2025/04/04 15:41:42.456 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:22, a: 688, b: 688, c: Nw9p, }"] [2025/04/04 15:41:42.556 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 689, b: 689, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:23, }"] [2025/04/04 15:41:42.656 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 690, b: 690, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:25, }"] [2025/04/04 15:41:42.756 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 691, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:26, a: 691, }"] [2025/04/04 15:41:42.856 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 692, b: 692, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:27, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:41:42.957 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:28, a: 693, b: 693, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:43.057 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:29, a: 694, b: 694, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:43.157 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:27, a: 695, b: 695, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] running resign_owner ***************** properties ***************** "threadcount"="2" "dotransactions"="false" "mysql.user"="root" "mysql.port"="4000" "table"="a4" "insertproportion"="0" "updateproportion"="0" "recordcount"="1000" "readallfields"="true" "mysql.host"="127.0.0.1" "workload"="core" "requestdistribution"="uniform" "mysql.db"="multi_rocks" "scanproportion"="0" "operationcount"="0" "readproportion"="0" ********************************************** [2025/04/04 15:41:43.257 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:31, a: 696, b: 696, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:43.357 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 697, b: 697, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:32, }"] pass test: resign owner cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:41:43 CST 2025] <<<<<< run test case cli_with_auth success! >>>>>> [2025/04/04 15:41:43.458 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 698, b: 698, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:33, }"] [2025/04/04 15:41:43.558 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 699, b: 699, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:57:01, }"] [2025/04/04 15:41:43.658 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:36, a: 700, b: 700, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:43.758 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:38, a: 702, b: 702, c: Nw9p, }"] [2025/04/04 15:41:43.858 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 703, b: 703, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:39, }"] Run finished, takes 948.722449ms INSERT - Takes(s): 0.9, Count: 1000, OPS: 1165.1, Avg(us): 1862, Min(us): 919, Max(us): 90272, 95th(us): 2000, 99th(us): 5000 [2025/04/04 15:41:43.959 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 704, b: 704, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:40, }"] [2025/04/04 15:41:44.059 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 705, b: 705, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:41, }"] [2025/04/04 15:41:44.159 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:42, a: 706, b: 706, c: Nw9p, }"] running get_tso check_redo_resolved_ts 7fe74488-ff74-4a3f-8d0c-cd20de57ea8e 457114254771224585 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/7fe74488-ff74-4a3f-8d0c-cd20de57ea8e/meta run task successfully [2025/04/04 15:41:44.259 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 707, b: 707, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:43, }"] [2025/04/04 15:41:44.359 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 708, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:44, a: 708, }"] pass test: get tso ***************** properties ***************** "mysql.db"="multi_rocks" "readproportion"="0" "threadcount"="2" "workload"="core" "mysql.user"="root" "recordcount"="1000" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "insertproportion"="0" "dotransactions"="false" "operationcount"="0" "mysql.port"="4000" "updateproportion"="0" "table"="a5" "readallfields"="true" "scanproportion"="0" ********************************************** [2025/04/04 15:41:44.459 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:45, a: 709, b: 709, }"] [2025/04/04 15:41:44.560 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 710, b: 710, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:46, }"] [2025/04/04 15:41:44.660 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:52:47, a: 711, b: 711, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:44.760 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 713, b: 713, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:49, }"] [2025/04/04 15:41:44.860 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 714, b: 714, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:50, }"] [2025/04/04 15:41:44.960 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 715, b: 715, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:51, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:41:45.061 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 716, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:52, a: 716, }"] [2025/04/04 15:41:45.161 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:53, a: 717, b: 717, c: Nw9p, }"] [2025/04/04 15:41:45.261 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 718, b: 718, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:54, }"] [2025/04/04 15:41:45.361 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 719, b: 719, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:55, }"] [2025/04/04 15:41:45.461 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 720, b: 720, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:57, }"] wait process cdc.test exit for 2-th time... running verify_table [2025/04/04 15:41:45.562 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:58, a: 721, b: 721, }"] [2025/04/04 15:41:45.662 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 722, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:52:59, a: 722, }"] pass test: verify table [2025/04/04 15:41:45.762 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:00, a: 723, b: 723, c: Nw9p, }"] [2025/04/04 15:41:45.862 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 724, b: 724, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:01, }"] [2025/04/04 15:41:45.962 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 725, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:02, a: 725, }"] Run finished, takes 1.028415259s INSERT - Takes(s): 1.0, Count: 1000, OPS: 991.3, Avg(us): 2011, Min(us): 893, Max(us): 90953, 95th(us): 2000, 99th(us): 24000 wait process cdc.test exit for 3-th time... [2025/04/04 15:41:46.063 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:03, a: 726, b: 726, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:46.163 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 727, b: 727, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:04, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + CDC_BINARY=cdc.test + SINK_TYPE=mysql + TEST_HOST_LIST=(UP_TIDB_HOST UP_PD_HOST_{1..3} UP_TIKV_HOST_{1..3}) + lan_addrs=($(hostname -I)) ++ hostname -I + lan_addr=10.233.78.216 + export UP_TIDB_HOST=10.233.78.216 UP_PD_HOST_1=10.233.78.216 UP_PD_HOST_2=10.233.78.216 UP_PD_HOST_3=10.233.78.216 UP_TIKV_HOST_1=10.233.78.216 UP_TIKV_HOST_2=10.233.78.216 UP_TIKV_HOST_3=10.233.78.216 + UP_TIDB_HOST=10.233.78.216 + UP_PD_HOST_1=10.233.78.216 + UP_PD_HOST_2=10.233.78.216 + UP_PD_HOST_3=10.233.78.216 + UP_TIKV_HOST_1=10.233.78.216 + UP_TIKV_HOST_2=10.233.78.216 + UP_TIKV_HOST_3=10.233.78.216 + proxy_pid= ++ shuf -i 10081-20081 -n1 + proxy_port=13258 + trap 'stop_tidb_cluster && stop_proxy' EXIT + '[' mysql == mysql ']' + prepare + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIDB_HOST + echo 'UP_TIDB_HOST 10.233.78.216' UP_TIDB_HOST 10.233.78.216 + case $(printenv $host) in ++ printenv UP_TIDB_HOST + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_PD_HOST_1 + echo 'UP_PD_HOST_1 10.233.78.216' UP_PD_HOST_1 10.233.78.216 + case $(printenv $host) in ++ printenv UP_PD_HOST_1 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_PD_HOST_2 + echo 'UP_PD_HOST_2 10.233.78.216' UP_PD_HOST_2 10.233.78.216 + case $(printenv $host) in ++ printenv UP_PD_HOST_2 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_PD_HOST_3 + echo 'UP_PD_HOST_3 10.233.78.216' UP_PD_HOST_3 10.233.78.216 + case $(printenv $host) in ++ printenv UP_PD_HOST_3 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIKV_HOST_1 + echo 'UP_TIKV_HOST_1 10.233.78.216' UP_TIKV_HOST_1 10.233.78.216 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_1 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIKV_HOST_2 + echo 'UP_TIKV_HOST_2 10.233.78.216' UP_TIKV_HOST_2 10.233.78.216 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_2 + for host in "${TEST_HOST_LIST[@]}" ++ printenv UP_TIKV_HOST_3 + echo 'UP_TIKV_HOST_3 10.233.78.216' UP_TIKV_HOST_3 10.233.78.216 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_3 + rm -rf /tmp/tidb_cdc_test/http_proxies + mkdir -p /tmp/tidb_cdc_test/http_proxies + stop_tidb_cluster cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2025/04/04 15:41:46.263 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 728, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:05, a: 728, }"] [2025/04/04 15:41:46.363 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 729, b: 729, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:06, }"] [2025/04/04 15:41:46.463 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 730, b: 730, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:07, }"] [2025/04/04 15:41:46.387 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:41:46.388 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:41:46.388 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:41:46.389 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:41:46.389 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114249305260065] [resolvedTs=457114255793586180] [2025/04/04 15:41:46.391 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:41:46.391 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:41:46.391 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:41:46.393 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=4.384408ms] [2025/04/04 15:41:46.397 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:41:46.400 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:41:46.403 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.403 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.403 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:41:46.404 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:41:46.405 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:41:46.405 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:41:46.405 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:41:46.405 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:41:46.405 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:41:46.405 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:41:46.416 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:41:46.419 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:41:46.419 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:41:46.564 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 731, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:08, a: 731, }"] [2025/04/04 15:41:46.664 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:09, a: 732, b: 732, }"] running create_changefeed_v2 [2025/04/04 15:41:46.764 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 733, b: 733, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:10, }"] [2025/04/04 15:41:46.864 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:11, a: 734, b: 734, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:46.964 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:12, a: 735, b: 735, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:47.064 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:13, a: 736, b: 736, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:47.165 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:14, a: 737, b: 737, c: Nw9p, }"] table multi_rocks.finish_mark not exists for 1-th check, retry later [2025/04/04 15:41:47.265 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:15, a: 738, b: 738, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:47.365 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 739, b: 739, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:16, }"] [2025/04/04 15:41:47.465 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:17, a: 740, b: 740, c: Nw9p, }"] pass test: create changefeed v2 [2025/04/04 15:41:47.566 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 742, b: 742, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:19, }"] [2025/04/04 15:41:47.666 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 743, b: 743, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:20, }"] [2025/04/04 15:41:47.766 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 744, b: 744, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:21, }"] [2025/04/04 15:41:47.866 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:22, a: 745, b: 745, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:47.966 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 746, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:23, a: 746, }"] [2025/04/04 15:41:48.067 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 747, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:24, a: 747, }"] [2025/04/04 15:41:48.167 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 748, b: 748, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:25, }"] [2025/04/04 15:41:48.416 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=2.027826293s] [2025/04/04 15:41:48.417 +08:00] [ERROR] [redo.go:272] ["ignore unsupported DDL"] [ddl="{\"StartTs\":457114249305260059,\"CommitTs\":457114249305260065,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_storage_file\",\"Table\":\"t1\",\"TableID\":118,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:41:48.267 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 749, b: 749, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:26, }"] [2025/04/04 15:41:48.367 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 750, b: 750, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:56:18, }"] [2025/04/04 15:41:48.467 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 751, b: 751, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:28, }"] running delete_changefeed_v2 [2025/04/04 15:41:48.567 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 752, b: 752, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:29, }"] [2025/04/04 15:41:48.668 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:30, a: 753, b: 753, c: Nw9p, }"] [2025/04/04 15:41:48.768 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:31, a: 754, b: 754, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:48.868 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 755, b: 755, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:32, }"] [2025/04/04 15:41:48.968 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 756, b: 756, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:33, }"] pass test: remove changefeed v2 [2025/04/04 15:41:49.068 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 757, b: 757, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:34, }"] [2025/04/04 15:41:49.169 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 758, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:35, a: 758, }"] [2025/04/04 15:41:49.269 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 759, b: 759, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:36, }"] + start_tidb_cluster --workdir /tmp/tidb_cdc_test/http_proxies The 1 times to try to start tidb cluster... [2025/04/04 15:41:49.369 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:37, a: 760, b: 760, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:49.469 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:38, a: 761, b: 761, c: Nw9p, }"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [2025/04/04 15:41:49.570 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 762, b: 762, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:39, }"] [2025/04/04 15:41:49.670 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 763, b: 763, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:40, }"] [2025/04/04 15:41:49.770 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:41, a: 764, b: 764, }"] [Pipeline] // cache [Pipeline] } [2025/04/04 15:41:49.810 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114255793586180] [2025/04/04 15:41:49.810 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114255793586180] [Pipeline] // dir table multi_rocks.finish_mark not exists for 2-th check, retry later [Pipeline] } [2025/04/04 15:41:49.870 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 765, b: 765, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:42, }"] [2025/04/04 15:41:49.971 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:43, a: 766, b: 766, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [Pipeline] // withCredentials [2025/04/04 15:41:49.911 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114255793586180] [2025/04/04 15:41:49.911 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114255793586180] [Pipeline] } running unsafe_apis [Pipeline] // timeout [Pipeline] } [2025/04/04 15:41:50.071 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 767, b: 767, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:44, }"] [2025/04/04 15:41:50.171 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 768, b: 768, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:45, }"] [2025/04/04 15:41:50.271 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 769, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:46, a: 769, }"] [Pipeline] // stage [Pipeline] } status code 200 pass test: show metadata status code 200 status code 200 pass test: delete service_gc_safepoint status code 500 pass test: resolve lock [Pipeline] // container [Pipeline] } [2025/04/04 15:41:50.371 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:47, a: 770, b: 770, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:50.472 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:48, a: 771, b: 771, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [2025/04/04 15:41:50.413 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114255793586180] [2025/04/04 15:41:50.413 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114255793586180] [2025/04/04 15:41:50.513 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=457114255793586180] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:41:50.513 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:41:50.514 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] Apply redo log successfully [Pipeline] // podTemplate [Pipeline] } [2025/04/04 15:41:50.572 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:49, a: 772, b: 772, }"] [2025/04/04 15:41:50.672 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 773, b: 773, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:50, }"] [2025/04/04 15:41:50.772 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:51, a: 774, b: 774, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [2025/04/04 15:41:50.872 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:52, a: 775, b: 775, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:50.972 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 776, b: 776, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:53, }"] [2025/04/04 15:41:51.073 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:53:54, a: 777, b: 777, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:51.173 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:55, a: 778, b: 778, }"] [2025/04/04 15:41:51.273 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 779, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:56, a: 779, }"] [2025/04/04 15:41:51.373 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 780, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:57, a: 780, }"] [2025/04/04 15:41:51.473 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 781, b: 781, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:58, }"] check diff successfully [Fri Apr 4 15:41:51 CST 2025] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> count(*) 5000 table multi_rocks.finish_mark not exists for 3-th check, retry later [2025/04/04 15:41:51.574 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:53:59, a: 782, b: 782, c: Nw9p, }"] [2025/04/04 15:41:51.674 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 783, b: 783, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:00, }"] [2025/04/04 15:41:51.774 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 784, b: 784, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:01, }"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [2025/04/04 15:41:51.874 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 785, b: 785, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:02, }"] [2025/04/04 15:41:51.974 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 786, b: 786, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:03, }"] [Pipeline] // dir [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // withCredentials [2025/04/04 15:41:52.075 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 787, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:04, a: 787, }"] [2025/04/04 15:41:52.175 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 788, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:05, a: 788, }"] [2025/04/04 15:41:52.275 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:06, a: 789, b: 789, }"] [Pipeline] } [Pipeline] // timeout [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // stage [2025/04/04 15:41:52.375 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 790, b: 790, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:07, }"] [2025/04/04 15:41:52.475 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:08, a: 791, b: 791, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [Pipeline] } [Pipeline] // container [Pipeline] } [2025/04/04 15:41:52.575 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 792, b: 792, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:09, }"] [2025/04/04 15:41:52.676 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 793, b: 793, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:10, }"] [2025/04/04 15:41:52.776 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 794, b: 794, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:11, }"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [2025/04/04 15:41:52.876 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 795, b: 795, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:12, }"] [2025/04/04 15:41:52.976 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 796, b: 796, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:13, }"] wait process cdc.test exit for 3-th time... [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2025/04/04 15:41:53.076 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:14, a: 797, b: 797, c: Nw9p, }"] [2025/04/04 15:41:53.177 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 798, b: 798, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:15, }"] [2025/04/04 15:41:53.277 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 799, b: 799, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:16, }"] [Pipeline] // stage [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:41:53.377 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 800, b: 800, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:17, }"] [2025/04/04 15:41:53.477 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 801, b: 801, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:18, }"] [2025/04/04 15:41:53.577 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:19, a: 802, b: 802, c: Nw9p, }"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:41:53 CST 2025] <<<<<< run test case http_api_tls_with_user_auth success! >>>>>> table multi_rocks.finish_mark exists check diff successfully [2025/04/04 15:41:53.678 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 803, b: 803, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:20, }"] [2025/04/04 15:41:53.778 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 804, b: 804, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:21, }"] [2025/04/04 15:41:53.878 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:22, a: 805, b: 805, c: Nw9p, }"] [2025/04/04 15:41:53.978 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:23, a: 806, b: 806, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:54.078 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 807, b: 807, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:24, }"] wait process cdc.test exit for 1-th time... [2025/04/04 15:41:54.179 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:25, a: 808, b: 808, c: Nw9p, }"] [2025/04/04 15:41:54.279 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 809, b: 809, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:26, }"] [2025/04/04 15:41:54.379 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:28, a: 810, b: 810, }"] [2025/04/04 15:41:54.479 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:29, a: 811, b: 811, c: Nw9p, }"] [2025/04/04 15:41:54.579 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 812, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:30, a: 812, }"] start tidb cluster in /tmp/tidb_cdc_test/http_proxies Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... [2025/04/04 15:41:54.679 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 813, b: 813, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:31, }"] [2025/04/04 15:41:54.780 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:32, a: 814, b: 814, c: Nw9p, }"] cdc.test: no process found wait process cdc.test exit for 3-th time... [2025/04/04 15:41:54.880 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 815, b: 815, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:33, }"] [2025/04/04 15:41:54.980 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:34, a: 816, b: 816, c: Nw9p, }"] [2025/04/04 15:41:55.080 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 817, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:35, a: 817, }"] process cdc.test already exit [Fri Apr 4 15:41:55 CST 2025] <<<<<< run test case multi_rocks success! >>>>>> [2025/04/04 15:41:55.180 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:36, a: 818, b: 818, }"] [2025/04/04 15:41:55.281 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 819, b: 819, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:37, }"] [2025/04/04 15:41:55.381 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:38, a: 820, b: 820, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:55.481 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 821, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:39, a: 821, }"] [2025/04/04 15:41:55.581 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:40, a: 822, b: 822, }"] [2025/04/04 15:41:55.681 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 823, b: 823, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:41, }"] [2025/04/04 15:41:55.781 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:42, a: 824, b: 824, c: Nw9p, }"] [2025/04/04 15:41:55.882 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 825, b: 825, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:43, }"] [2025/04/04 15:41:55.982 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:44, a: 826, b: 826, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:56.082 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 827, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:45, a: 827, }"] [2025/04/04 15:41:56.182 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 828, b: 828, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:46, }"] [2025/04/04 15:41:56.282 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 829, b: 829, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:47, }"] [2025/04/04 15:41:56.383 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 830, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:48, a: 830, }"] [2025/04/04 15:41:56.483 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 831, b: 831, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:49, }"] [2025/04/04 15:41:56.583 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 832, b: 832, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:50, }"] start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:41:56.683 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 833, b: 833, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:51, }"] [2025/04/04 15:41:56.783 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 834, b: 834, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:52, }"] [2025/04/04 15:41:56.131 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:41:56.133 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:41:56.133 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=0386c122-8e01-4b7e-bc6e-15406a3750e2] [2025/04/04 15:41:56.134 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:41:56.134 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2025/04/04 15:41:56.246 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=267eab32-43d3-4f93-94aa-2e4de3887bf3] [captureID=0386c122-8e01-4b7e-bc6e-15406a3750e2] [processorDetail="{\"table_ids\":[114,116]}"] [2025/04/04 15:41:56.395 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=267eab32-43d3-4f93-94aa-2e4de3887bf3] [captureID=9cd74e7f-52d8-447d-85c9-9d8eef346d73] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:41:56.497 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=267eab32-43d3-4f93-94aa-2e4de3887bf3] [captureID=f49e559b-b008-4451-9fd3-0bd465864a43] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:41:56.497 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"0386c122-8e01-4b7e-bc6e-15406a3750e2\":[{\"ID\":114,\"Changefeed\":\"267eab32-43d3-4f93-94aa-2e4de3887bf3\"},{\"ID\":116,\"Changefeed\":\"267eab32-43d3-4f93-94aa-2e4de3887bf3\"}],\"9cd74e7f-52d8-447d-85c9-9d8eef346d73\":[],\"f49e559b-b008-4451-9fd3-0bd465864a43\":[]}"] [2025/04/04 15:41:56.497 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=267eab32-43d3-4f93-94aa-2e4de3887bf3&target-cp-id=f49e559b-b008-4451-9fd3-0bd465864a43&table-id=114"] [2025/04/04 15:41:56.594 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=114] [2025/04/04 15:41:56.595 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=267eab32-43d3-4f93-94aa-2e4de3887bf3&target-cp-id=f49e559b-b008-4451-9fd3-0bd465864a43&table-id=116"] [2025/04/04 15:41:56.645 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=116] [2025/04/04 15:41:56.645 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=0386c122-8e01-4b7e-bc6e-15406a3750e2] [targetCapture=f49e559b-b008-4451-9fd3-0bd465864a43] table move_table.check1 exists check diff successfully [2025/04/04 15:41:56.883 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:54:53, a: 835, b: 835, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:56.984 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:54, a: 836, b: 836, }"] [2025/04/04 15:41:57.084 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 837, b: 837, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:55, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:41:57.184 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 838, b: 838, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:56, }"] [2025/04/04 15:41:57.284 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:57, a: 839, b: 839, }"] [2025/04/04 15:41:57.385 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 840, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:58, a: 840, }"] [2025/04/04 15:41:57.485 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 841, b: 841, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:54:59, }"] [2025/04/04 15:41:57.585 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:00, a: 842, b: 842, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:57.685 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 843, b: 843, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:01, }"] [2025/04/04 15:41:57.785 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:02, a: 844, b: 844, c: Nw9p, }"] [2025/04/04 15:41:57.886 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 845, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:03, a: 845, }"] [2025/04/04 15:41:57.986 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:04, a: 846, b: 846, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:58.086 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:05, a: 847, b: 847, c: Nw9p, }"] [2025/04/04 15:41:58.186 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 848, b: 848, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:06, }"] [2025/04/04 15:41:58.286 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 849, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:07, a: 849, }"] [2025/04/04 15:41:58.387 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:08, a: 850, b: 850, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:58.487 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 851, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:09, a: 851, }"] [2025/04/04 15:41:58.587 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 852, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:10, a: 852, }"] [2025/04/04 15:41:58.687 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 853, b: 853, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:11, }"] [2025/04/04 15:41:58.787 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 854, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:12, a: 854, }"] [2025/04/04 15:41:58.887 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:13, a: 855, b: 855, c: Nw9p, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.78.216' (111) [2025/04/04 15:41:58.988 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 857, b: 857, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:15, }"] [2025/04/04 15:41:59.088 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 858, b: 858, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:16, }"] [2025/04/04 15:41:59.188 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:17, a: 859, b: 859, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:59.288 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:18, a: 860, b: 860, }"] [row2="{ a: 860, b: 860, c: MiqtEL, zFzkK: OaFy3Zx1EQDTUP5Sznp28zXVpCkD9MTfwD38, d: 15:55:18, }"] [2025/04/04 15:41:59.388 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 861, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:19, a: 861, }"] [row2="{ a: 861, b: 861, c: 49C2VVvi, zFzkK: yntFdKVFK4rgVXFqA57i8KY6ma1yHWu, d: 15:55:19, }"] [2025/04/04 15:41:59.489 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:20, a: 862, b: 862, c: Nw9p, }"] [2025/04/04 15:41:59.589 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 863, b: 863, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:21, }"] [row2="{ d: 15:55:21, a: 863, b: 863, c: O, zFzkK: rHJZKjpce60et1tVDVPIy0euKeL6qcBDgAujA1SYne4Y, }"] [2025/04/04 15:41:59.690 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:22, a: 864, b: 864, }"] [row2="{ a: 864, b: 864, c: fKPn, zFzkK: 4HBYibFzbcX02KQed, d: 15:55:22, }"] [2025/04/04 15:41:59.790 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 865, b: 865, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:23, }"] [row2="{ a: 865, b: 865, c: GvG4o, zFzkK: RuolP13i6jQTyk62GeEx, d: 15:55:23, }"] [2025/04/04 15:41:59.890 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:55:24, a: 866, b: 866, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: YaCH, zFzkK: aFZGI4KTMMo9xPqOVtPLzHaGVc7, d: 15:55:24, a: 866, b: 866, }"] [2025/04/04 15:41:59.990 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:25, a: 867, b: 867, }"] [row2="{ d: 15:55:25, a: 867, b: 867, c: s9mel4, zFzkK: FknG2, }"] [2025/04/04 15:42:00.090 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 868, b: 868, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:26, }"] [row2="{ zFzkK: owsnW1, d: 15:55:26, a: 868, b: 868, c: YBRngUk, }"] [2025/04/04 15:42:00.190 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:55:27, a: 869, b: 869, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ b: 869, c: 2wysuknc2, zFzkK: QCTVfNodIzlBnf9Di9zIn2a, d: 15:55:27, a: 869, }"] [2025/04/04 15:42:00.291 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 870, b: 870, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:28, }"] [2025/04/04 15:42:00.391 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 871, b: 871, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:29, }"] [2025/04/04 15:42:00.491 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 872, b: 872, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:30, }"] [2025/04/04 15:42:00.591 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:31, a: 873, b: 873, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:42:00.691 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 874, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:32, a: 874, }"] [2025/04/04 15:42:00.792 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:33, a: 875, b: 875, }"] [2025/04/04 15:42:00.892 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 876, b: 876, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:34, }"] [2025/04/04 15:42:00.992 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:35, a: 877, b: 877, c: Nw9p, }"] [2025/04/04 15:42:01.092 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 878, b: 878, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:36, }"] [2025/04/04 15:42:01.192 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:37, a: 879, b: 879, }"] [2025/04/04 15:42:01.293 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 880, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:38, a: 880, }"] [2025/04/04 15:42:01.393 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:39, a: 881, b: 881, }"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.78.216' (111) [2025/04/04 15:42:01.493 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 882, b: 882, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:40, }"] [2025/04/04 15:42:01.593 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 883, b: 883, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:41, }"] [2025/04/04 15:42:01.693 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 884, b: 884, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:42, }"] [2025/04/04 15:42:01.794 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:43, a: 885, b: 885, }"] [2025/04/04 15:42:01.894 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ d: 15:55:44, a: 886, b: 886, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [2025/04/04 15:41:51.896 +08:00] [INFO] [main.go:86] ["running ddl test: 0 createDropSchemaDDL"] [2025/04/04 15:41:52.210 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:41:52.210 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:41:52.572 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:41:52.574 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:41:52.575 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:41:52.578 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:41:52.853 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:41:52.926 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:41:53.233 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:41:53.235 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:41:53.253 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:41:53.259 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:41:53.587 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:41:53.599 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:41:53.854 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:41:53.857 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:41:53.861 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:41:53.947 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:41:54.253 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:41:54.261 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:41:54.691 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:41:54.699 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:41:54.704 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:41:54.712 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:41:55.092 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:41:55.111 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:41:55.379 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:41:55.395 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:41:55.402 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:41:55.414 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:41:55.750 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:41:55.771 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:41:56.118 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:41:56.129 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:41:56.136 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:41:56.147 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:41:56.425 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:41:56.436 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:41:56.836 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:41:56.853 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:41:56.854 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:41:56.865 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [mysql] 2025/04/04 15:41:56 connection.go:299: invalid connection [2025/04/04 15:42:01.924 +08:00] [INFO] [main.go:86] ["running ddl test: 1 truncateDDL"] [2025/04/04 15:42:01.994 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 887, b: 887, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:45, }"] [2025/04/04 15:42:02.094 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:46, a: 888, b: 888, }"] [2025/04/04 15:42:02.194 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 889, b: 889, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:47, }"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2025/04/04 15:42:02.172 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:42:02.176 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:42:01.572 +08:00] [INFO] [main.go:61] ["table mover started"] [2025/04/04 15:42:01.574 +08:00] [INFO] [main.go:165] ["new cluster initialized"] [2025/04/04 15:42:01.574 +08:00] [DEBUG] [main.go:191] ["retrieved owner ID"] [ownerID=0386c122-8e01-4b7e-bc6e-15406a3750e2] [2025/04/04 15:42:01.574 +08:00] [DEBUG] [main.go:198] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2025/04/04 15:42:01.575 +08:00] [DEBUG] [main.go:209] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2025/04/04 15:42:01.695 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=267eab32-43d3-4f93-94aa-2e4de3887bf3] [captureID=0386c122-8e01-4b7e-bc6e-15406a3750e2] [processorDetail="{\"table_ids\":[118]}"] [2025/04/04 15:42:01.845 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=267eab32-43d3-4f93-94aa-2e4de3887bf3] [captureID=9cd74e7f-52d8-447d-85c9-9d8eef346d73] [processorDetail="{\"table_ids\":[]}"] [2025/04/04 15:42:01.994 +08:00] [DEBUG] [main.go:228] ["retrieved processor details"] [changefeed=267eab32-43d3-4f93-94aa-2e4de3887bf3] [captureID=f49e559b-b008-4451-9fd3-0bd465864a43] [processorDetail="{\"table_ids\":[116]}"] [2025/04/04 15:42:01.994 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"0386c122-8e01-4b7e-bc6e-15406a3750e2\":[{\"ID\":118,\"Changefeed\":\"267eab32-43d3-4f93-94aa-2e4de3887bf3\"}],\"9cd74e7f-52d8-447d-85c9-9d8eef346d73\":[],\"f49e559b-b008-4451-9fd3-0bd465864a43\":[{\"ID\":116,\"Changefeed\":\"267eab32-43d3-4f93-94aa-2e4de3887bf3\"}]}"] [2025/04/04 15:42:01.994 +08:00] [DEBUG] [main.go:287] ["preparing HTTP API call to owner"] [formStr="cf-id=267eab32-43d3-4f93-94aa-2e4de3887bf3&target-cp-id=9cd74e7f-52d8-447d-85c9-9d8eef346d73&table-id=116"] [2025/04/04 15:42:02.045 +08:00] [INFO] [main.go:179] ["moved table successful"] [tableID=116] [2025/04/04 15:42:02.045 +08:00] [INFO] [main.go:113] ["all tables are moved"] [sourceCapture=f49e559b-b008-4451-9fd3-0bd465864a43] [targetCapture=9cd74e7f-52d8-447d-85c9-9d8eef346d73] check diff successfully [2025/04/04 15:42:02.294 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:48, a: 890, b: 890, }"] [2025/04/04 15:42:02.395 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:49, a: 891, b: 891, c: Nw9p, }"] Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:42:02.386 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:42:02.387 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:42:02.388 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:42:02.390 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:42:02.495 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:50, a: 892, b: 892, }"] [2025/04/04 15:42:02.595 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:51, a: 893, b: 893, }"] [2025/04/04 15:42:02.695 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:52, a: 894, b: 894, }"] [2025/04/04 15:42:02.621 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:42:02.634 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] table move_table.check2 exists check diff successfully [2025/04/04 15:42:02.795 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:53, a: 895, b: 895, c: Nw9p, }"] [2025/04/04 15:42:02.896 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 896, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:54, a: 896, }"] [2025/04/04 15:42:02.824 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:42:02.825 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:42:02.861 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:42:02.867 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:42:03.027 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:42:02.996 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:55, a: 897, b: 897, c: Nw9p, }"] [2025/04/04 15:42:03.096 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ b: 898, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:56, a: 898, }"] [2025/04/04 15:42:03.196 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ a: 899, b: 899, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:57, }"] [2025/04/04 15:42:03.079 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:42:03.217 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:42:03.218 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:42:03.269 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:42:03.282 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] wait process cdc.test exit for 1-th time... [2025/04/04 15:42:03.296 +08:00] [WARN] [diff.go:942] ["target lack data"] [row="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:55:58, a: 900, b: 900, c: Nw9p, }"] [2025/04/04 15:42:03.397 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:55:59, a: 901, b: 901, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 901, b: 901, c: ajXGSHBA, zFzkK: sLjGVG6JcAzQ5Zn089Od0kk, d: 15:55:59, }"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360025605239323 TiDB Cluster ID. tikv_gc_leader_uuid 657feff99ac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:7694, start at 2025-04-04 15:42:01.334052005 +0800 CST m=+2.317343714 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:01.342 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:01.342 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:01.342 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360025605239323 TiDB Cluster ID. tikv_gc_leader_uuid 657feff99ac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:7694, start at 2025-04-04 15:42:01.334052005 +0800 CST m=+2.317343714 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:01.342 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:01.342 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:01.342 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360020460204103 TiDB Cluster ID. tikv_gc_leader_uuid 657feff9ac00006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:7763, start at 2025-04-04 15:42:01.433536017 +0800 CST m=+2.344764409 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:01.442 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:01.442 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:01.442 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_proxies/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_proxies/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["10.233.78.216:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:42:03.403 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:42:03.469 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/cli_tls_with_auth/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:42:03.497 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:56:00, a: 902, b: 902, c: Nw9p, }"] [row2="{ d: 15:56:00, a: 902, b: 902, c: gWb, zFzkK: qv5nL49TP9rvhiALVPpC3B6sC, }"] [2025/04/04 15:42:03.597 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 903, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:56:01, a: 903, }"] [row2="{ zFzkK: luwgIvsEzSGWo18u3KbIdtq4X, d: 15:56:01, a: 903, b: 903, c: E5WeEt, }"] [2025/04/04 15:42:03.697 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ b: 904, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:56:02, a: 904, }"] [row2="{ b: 904, c: KxAJCpQ, zFzkK: Wa1B5gkVwLy5gj5Ds5wGynvJgVMNkzC9vef7tMQ6, d: 15:56:02, a: 904, }"] [2025/04/04 15:42:03.607 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:42:03.610 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:42:03.656 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:42:03.677 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:42:03.798 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:42:03.797 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 905, b: 905, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:56:03, }"] [row2="{ d: 15:56:03, a: 905, b: 905, c: FY, zFzkK: 1lzSTpU, }"] [2025/04/04 15:42:03.897 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:56:04, a: 906, b: 906, c: Nw9p, }"] [row2="{ a: 906, b: 906, c: EhxWxW, zFzkK: 8D, d: 15:56:04, }"] wait process cdc.test exit for 2-th time... [2025/04/04 15:42:03.844 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:42:03.992 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:42:04.008 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:42:04.041 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:42:04.071 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:42:03.998 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ a: 907, b: 907, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, d: 15:56:05, }"] [row2="{ b: 907, c: RzGJv4Zjm, zFzkK: dYHlVJwl5cOOM, d: 15:56:05, a: 907, }"] [2025/04/04 15:42:04.098 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:56:06, a: 908, b: 908, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ c: orsPHCB8BG, zFzkK: WJIWurc4WxKc3GZTK1, d: 15:56:06, a: 908, b: 908, }"] [2025/04/04 15:42:04.198 +08:00] [WARN] [diff.go:937] ["find different row"] [column=c] [row1="{ d: 15:56:07, a: 909, b: 909, c: Nw9p, zFzkK: vC63MzfcpTEvSC8cdGKUmrr1nJNbKLAZ8PnNOz, }"] [row2="{ a: 909, b: 909, c: V5IR, zFzkK: J64dNa1eM3SbeiyVMtBcyTALdZZStQmk6t3nN, d: 15:56:07, }"] [2025/04/04 15:42:04.187 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:42:04.237 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:04 CST 2025] <<<<<< run test case move_table success! >>>>>> [2025/04/04 15:42:04.380 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:42:04.404 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:42:04.429 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:42:04.468 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:42:04.568 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:42:04.620 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:42:04.765 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:42:04.795 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:42:04.814 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:42:04.850 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:42:04.970 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:42:05.015 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/ci_collation_compatibility/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2025/04/04 15:42:05.163 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:42:05.195 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:42:05.205 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:42:05.243 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:42:05.357 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:42:05.402 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:42:05.546 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:42:05.579 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:42:05.588 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:42:05.626 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:42:05.744 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:42:05.791 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] + start_proxy + echo 'dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log...' dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log... + proxy_pid=8996 + echo 'proxy port: 13258' proxy port: 13258 + echo 'proxy pid: 8996' proxy pid: 8996 + sleep 5 + GO111MODULE=on + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + go run /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go --port=13258 go: downloading go1.23.4 (linux/amd64) [2025/04/04 15:42:05.935 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:42:05.969 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:42:05.990 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:42:06.028 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:42:06.121 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:42:06.173 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:42:06.308 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:42:06.341 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:42:06.360 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:42:06.407 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:42:06.503 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] [2025/04/04 15:42:06.559 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360041812492062 TiDB Cluster ID. tikv_gc_leader_uuid 657feffca34002d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:13184, start at 2025-04-04 15:42:04.43854944 +0800 CST m=+2.189481286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:04.445 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:04.429 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:04.429 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360041812492062 TiDB Cluster ID. tikv_gc_leader_uuid 657feffca34002d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:13184, start at 2025-04-04 15:42:04.43854944 +0800 CST m=+2.189481286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:04.445 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:04.429 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:04.429 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360041661182378 TiDB Cluster ID. tikv_gc_leader_uuid 657feffcc2c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:13251, start at 2025-04-04 15:42:04.568264281 +0800 CST m=+2.224872359 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:04.577 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:04.555 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:04.555 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:42:06.709 +08:00] [INFO] [main.go:221] ["0 insert success: 2400"] [2025/04/04 15:42:06.753 +08:00] [INFO] [main.go:235] ["0 delete success: 1200"] [2025/04/04 15:42:06.758 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] [2025/04/04 15:42:06.809 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] [2025/04/04 15:42:06.908 +08:00] [INFO] [main.go:221] ["0 insert success: 2500"] Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release start tidb cluster in /tmp/tidb_cdc_test/ci_collation_compatibility Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14478.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... + set +x + tso='457114262978428929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114262978428929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:10 CST 2025] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1450514507.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading google.golang.org/grpc v1.64.0 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/pingcap/log v1.1.1-0.20241212030209-7e3ff8601a2a go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading golang.org/x/net v0.33.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading golang.org/x/sys v0.28.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/gorilla/websocket v1.5.1 + export http_proxy=http://127.0.0.1:13258 + http_proxy=http://127.0.0.1:13258 + export https_proxy=http://127.0.0.1:13258 + https_proxy=http://127.0.0.1:13258 + echo 'try to connect pd cluster via proxy, pd addr: 10.233.78.216:2379' try to connect pd cluster via proxy, pd addr: 10.233.78.216:2379 + ensure 10 curl http://10.233.78.216:2379/ curl http://10.233.78.216:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 13258: Connection refused run task failed 1-th time, retry later go: downloading golang.org/x/text v0.21.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 go: downloading google.golang.org/protobuf v1.35.2 go: downloading google.golang.org/genproto v0.0.0-20240401170217-c3f982113cda Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:42:11.985 +08:00] [INFO] [main.go:86] ["running ddl test: 2 addDropColumnDDL"] [2025/04/04 15:42:12.202 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:42:12.210 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) curl http://10.233.78.216:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 13258: Connection refused run task failed 2-th time, retry later [2025/04/04 15:42:12.419 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:42:12.422 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:42:12.428 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:42:12.430 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:42:12.630 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:42:12.650 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:42:12.862 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:42:12.864 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:42:12.898 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:42:12.900 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:42:13.048 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:42:13.103 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:42:13.238 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:42:13.240 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:42:13.299 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:42:13.301 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:42:13.419 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:42:13.496 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:42:13.606 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:42:13.608 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:13 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/002b1fa1-06de-4f63-ae8e-b11732aa1dde {"id":"002b1fa1-06de-4f63-ae8e-b11732aa1dde","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752530} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfd3fde8 002b1fa1-06de-4f63-ae8e-b11732aa1dde /tidb/cdc/default/default/upstream/7489360041812492062 {"id":7489360041812492062,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/002b1fa1-06de-4f63-ae8e-b11732aa1dde {"id":"002b1fa1-06de-4f63-ae8e-b11732aa1dde","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752530} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfd3fde8 002b1fa1-06de-4f63-ae8e-b11732aa1dde /tidb/cdc/default/default/upstream/7489360041812492062 {"id":7489360041812492062,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/002b1fa1-06de-4f63-ae8e-b11732aa1dde {"id":"002b1fa1-06de-4f63-ae8e-b11732aa1dde","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752530} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbfd3fde8 002b1fa1-06de-4f63-ae8e-b11732aa1dde /tidb/cdc/default/default/upstream/7489360041812492062 {"id":7489360041812492062,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14566.out cli changefeed create --start-ts=457114262978428929 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [2025/04/04 15:42:13.704 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:42:13.707 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:42:13.795 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:42:13.915 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:42:13.318 +08:00] [WARN] [diff.go:766] ["rows is not equal"] [table=`test`.`itest`] [where="((TRUE) AND TRUE)"] [cost=1m33.769693151s] [2025/04/04 15:42:13.321 +08:00] [WARN] [diff.go:384] ["check chunk data not equal"] [chunk="{\"id\":0,\"bounds\":[],\"where\":\"((TRUE) AND TRUE)\",\"args\":null,\"state\":\"failed\"}"] Create changefeed successfully! ID: e2719cc1-4149-4240-aefb-6c4c67762cd7 Info: {"upstream_id":7489360041812492062,"namespace":"default","id":"e2719cc1-4149-4240-aefb-6c4c67762cd7","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:42:14.073153376+08:00","start_ts":457114262978428929,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114262978428929,"checkpoint_ts":457114262978428929,"checkpoint_time":"2025-04-04 15:42:09.062"} PASS [2025/04/04 15:42:13.977 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:42:13.979 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:42:14.125 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:42:14.127 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:42:14.170 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] Starting Upstream TiDB... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360069113822555 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0046c40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:12709, start at 2025-04-04 15:42:12.404261769 +0800 CST m=+2.546790234 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:12.412 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:12.401 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:12.401 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360069113822555 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0046c40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:12709, start at 2025-04-04 15:42:12.404261769 +0800 CST m=+2.546790234 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:12.412 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:12.401 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:12.401 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360075098670550 TiDB Cluster ID. tikv_gc_leader_uuid 657ff00482c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:12782, start at 2025-04-04 15:42:12.514956703 +0800 CST m=+2.585090487 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:12.523 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:12.491 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:12.491 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:42:14.311 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:42:14.350 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:42:14.352 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:42:14.509 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:42:14.511 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:42:14.548 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:42:14.706 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:42:14.748 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:42:14.749 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:42:14.918 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:42:14.920 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:42:14.965 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:42:15.136 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:42:15.174 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:42:15.177 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:42:15.332 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:42:15.334 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:42:15.368 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] ____________________________________ *************************** 1. row *************************** primary_ts: 457114252738560000 secondary_ts: 457114252979208240 primary_ts is 457114252738560000 start_ts is 457114245284233218 check successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:09 CST 2025] <<<<<< run test case syncpoint_check_ts success! >>>>>> [2025/04/04 15:42:15.534 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:42:15.581 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:42:15.583 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:42:15.728 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:42:15.730 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] + set +x table savepoint.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:42:15.775 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:42:15.937 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:42:15.992 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:42:15.994 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:42:16.172 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:42:16.174 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:42:16.236 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:42:16.377 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:42:16.448 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:42:16.450 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] curl http://10.233.78.216:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 13258: Connection refused run task failed 3-th time, retry later [2025/04/04 15:42:16.630 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:42:16.633 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:42:16.700 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [Fri Apr 4 15:42:16 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1400714009.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:42:16.851 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] [2025/04/04 15:42:16.933 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] [2025/04/04 15:42:16.935 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table savepoint.finish_mark not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled False If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360087959196804 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0080fc000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:27145, start at 2025-04-04 15:42:16.169010609 +0800 CST m=+2.339109832 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:16.176 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:16.178 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:16.178 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled False If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360087959196804 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0080fc000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:27145, start at 2025-04-04 15:42:16.169010609 +0800 CST m=+2.339109832 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:16.176 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:16.178 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:16.178 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled False If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360092935766246 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0085a4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-0vf91-8tdb4-6lzwr, pid:27223, start at 2025-04-04 15:42:16.463039173 +0800 CST m=+2.571111763 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:16.471 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:16.471 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:16.471 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/ci_collation_compatibility/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/ci_collation_compatibility/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360096210994760 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0092b80010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:39405, start at 2025-04-04 15:42:17.284441168 +0800 CST m=+2.879924306 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:17.293 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:17.262 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:17.262 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360096210994760 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0092b80010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:39405, start at 2025-04-04 15:42:17.284441168 +0800 CST m=+2.879924306 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:17.293 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:17.262 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:17.262 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360078067331138 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0092b4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:39465, start at 2025-04-04 15:42:17.309136253 +0800 CST m=+2.822662823 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:17.360 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:17.361 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:17.361 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/cli_tls_with_auth/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:42:18.961 +08:00] [WARN] [diff.go:557] ["checksum is not equal"] [table=`test`.`ntest`] [where="((TRUE) AND TRUE)"] [sourceChecksum=0] [targetChecksum=11832841] [getSourceChecksumCost=2.56167ms] [getTargetChecksumCost=13.347652ms] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:19 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6d7d8713-c3bd-4131-9831-35c3abc8305d {"id":"6d7d8713-c3bd-4131-9831-35c3abc8305d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752536} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbff38fd9 6d7d8713-c3bd-4131-9831-35c3abc8305d /tidb/cdc/default/default/upstream/7489360069113822555 {"id":7489360069113822555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6d7d8713-c3bd-4131-9831-35c3abc8305d {"id":"6d7d8713-c3bd-4131-9831-35c3abc8305d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752536} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbff38fd9 6d7d8713-c3bd-4131-9831-35c3abc8305d /tidb/cdc/default/default/upstream/7489360069113822555 {"id":7489360069113822555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6d7d8713-c3bd-4131-9831-35c3abc8305d {"id":"6d7d8713-c3bd-4131-9831-35c3abc8305d","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752536} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbff38fd9 6d7d8713-c3bd-4131-9831-35c3abc8305d /tidb/cdc/default/default/upstream/7489360069113822555 {"id":7489360069113822555,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table savepoint.finish_mark exists check diff successfully ***************** properties ***************** "fieldcount"="1000" "dotransactions"="false" "scanproportion"="0" "requestdistribution"="uniform" "operationcount"="0" "insertproportion"="0" "readallfields"="true" "mysql.port"="4000" "fieldlengthdistribution"="constant" "mysql.host"="127.0.0.1" "fieldlength"="15000" "updateproportion"="0" "mysql.db"="consistent_replicate_storage_file_large_value" "mysql.user"="root" "recordcount"="100" "readproportion"="0" "threadcount"="10" "workload"="core" ********************************************** + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ci_collation_compatibility.cli.28532.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 1-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/cli_tls_with_auth Starting TLS PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... + '[' mysql == mysql ']' + check_table_exists test.t2 127.0.0.1 3306 table test.t2 not exists for 1-th check, retry later table test.t2 exists + sleep 10 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:42:21 CST 2025] <<<<<< run test case savepoint success! >>>>>> + set +x + tso='457114266023231489 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114266023231489 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:22 CST 2025] <<<<<< START cdc server in ci_collation_compatibility case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ci_collation_compatibility.2856428566.out server --log-file /tmp/tidb_cdc_test/ci_collation_compatibility/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ci_collation_compatibility/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:42:22.161 +08:00] [INFO] [main.go:86] ["running ddl test: 3 addDropColumnDDL2"] [2025/04/04 15:42:22.379 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:42:22.379 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:42:22.585 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:42:22.587 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:42:22.590 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:42:22.591 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:42:22.793 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:42:22.797 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:42:22.986 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:42:22.987 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:42:22.991 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:42:22.992 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] curl http://10.233.78.216:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to 127.0.0.1 port 13258: Connection refused run task failed 4-th time, retry later [2025/04/04 15:42:23.181 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:42:23.195 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:42:23.370 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:42:23.371 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:42:23.383 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:42:23.384 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore [2025/04/04 15:42:23.554 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:42:23.573 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:42:23.745 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:42:23.746 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:42:23.763 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:42:23.764 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:42:23.945 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:42:23.959 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] time="2025-04-04T15:42:23+08:00" level=info msg="Listening on 127.0.0.1:13258" time="2025-04-04T15:42:23+08:00" level=info msg="Not intercepting TLS connections" [2025/04/04 15:42:24.126 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:42:24.126 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:42:24.137 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:42:24.138 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:42:24.315 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:42:24.320 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:42:24.504 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:42:24.505 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:42:24.511 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:42:24.511 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:42:24.718 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:42:24.723 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] start tidb cluster in /tmp/tidb_cdc_test/server_config_compatibility Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Starting TLS TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:42:24.921 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:42:24.922 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:42:24.922 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:42:24.923 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:42:25.105 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:42:25.117 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:42:25.283 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:42:25.284 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:42:25.312 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:42:25.313 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:25 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a03d6918-ebd3-43a7-9546-b21c358c4c5e {"id":"a03d6918-ebd3-43a7-9546-b21c358c4c5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752542} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0012ceb a03d6918-ebd3-43a7-9546-b21c358c4c5e /tidb/cdc/default/default/upstream/7489360087959196804 {"id":7489360087959196804,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a03d6918-ebd3-43a7-9546-b21c358c4c5e {"id":"a03d6918-ebd3-43a7-9546-b21c358c4c5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752542} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0012ceb a03d6918-ebd3-43a7-9546-b21c358c4c5e /tidb/cdc/default/default/upstream/7489360087959196804 {"id":7489360087959196804,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a03d6918-ebd3-43a7-9546-b21c358c4c5e {"id":"a03d6918-ebd3-43a7-9546-b21c358c4c5e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752542} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0012ceb a03d6918-ebd3-43a7-9546-b21c358c4c5e /tidb/cdc/default/default/upstream/7489360087959196804 {"id":7489360087959196804,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ci_collation_compatibility.cli.28618.out cli changefeed create --start-ts=457114266023231489 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?safe-mode=true' [2025/04/04 15:42:25.483 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:42:25.508 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] Create changefeed successfully! ID: b5897c2c-6b31-447b-ad2e-c2d2c5da8d94 Info: {"upstream_id":7489360087959196804,"namespace":"default","id":"b5897c2c-6b31-447b-ad2e-c2d2c5da8d94","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:42:25.719208153+08:00","start_ts":457114266023231489,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114266023231489,"checkpoint_ts":457114266023231489,"checkpoint_time":"2025-04-04 15:42:20.677"} PASS [2025/04/04 15:42:25.669 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:42:25.670 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:42:25.690 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:42:25.691 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:42:25.854 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:42:25.876 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:42:26.037 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:42:26.039 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:42:26.059 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:42:26.060 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:42:26.228 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:42:26.240 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:42:26.419 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:42:26.420 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:42:26.430 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:42:26.431 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:42:26.623 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] [2025/04/04 15:42:26.625 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:42:26.842 +08:00] [INFO] [main.go:235] ["0 delete success: 1200"] [2025/04/04 15:42:26.843 +08:00] [INFO] [main.go:221] ["0 insert success: 2400"] [2025/04/04 15:42:26.850 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] [2025/04/04 15:42:26.851 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] Starting TLS TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:42:27.048 +08:00] [INFO] [main.go:221] ["1 insert success: 2500"] [2025/04/04 15:42:27.048 +08:00] [INFO] [main.go:221] ["0 insert success: 2500"] + set +x + run_sql 'SELECT * from test.t2' 127.0.0.1 3306 + check_contains 'a: 2023-02-08' + stop_tidb_cluster Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360132260082184 TiDB Cluster ID. tikv_gc_leader_uuid 657ff011e280003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:28515, start at 2025-04-04 15:42:26.189547635 +0800 CST m=+2.480195197 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:26.199 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:26.186 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:26.186 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360132260082184 TiDB Cluster ID. tikv_gc_leader_uuid 657ff011e280003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:28515, start at 2025-04-04 15:42:26.189547635 +0800 CST m=+2.480195197 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:26.199 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:26.186 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:26.186 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360126236631121 TiDB Cluster ID. tikv_gc_leader_uuid 657ff011bd4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:28584, start at 2025-04-04 15:42:26.066942281 +0800 CST m=+2.286151764 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:26.074 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:26.037 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:26.037 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360145989388168 TiDB Cluster ID. tikv_gc_leader_uuid 657ff013cf0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:41067, start at 2025-04-04 15:42:28.17616565 +0800 CST m=+1.602717253 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:28.183 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:28.156 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:28.156 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + pd_host=127.0.0.1 + pd_port=2579 + is_tls=true + '[' true == true ']' ++ run_cdc_cli tso query --pd=https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41109.out cli tso query --pd=https://127.0.0.1:2579 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Run finished, takes 9.170591172s INSERT_ERROR - Takes(s): 8.3, Count: 100, OPS: 12.1, Avg(us): 296281, Min(us): 123151, Max(us): 514543, 95th(us): 482000, 99th(us): 503000 table consistent_replicate_storage_file_large_value.usertable exists table consistent_replicate_storage_file_large_value.check1 not exists for 1-th check, retry later [Fri Apr 4 15:42:30 CST 2025] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.2980629808.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) curl http://10.233.78.216:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 19 100 19 0 0 19000 0 --:--:-- --:--:-- --:--:-- 19000 404 page not found run task successfully + echo started proxy pid: 8996 started proxy pid: 8996 + echo started proxy at port: 13258 started proxy at port: 13258 + cd /tmp/tidb_cdc_test/http_proxies ++ run_cdc_cli_tso_query 10.233.78.216 2379 + pd_host=10.233.78.216 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://10.233.78.216:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.13013.out cli tso query --pd=http://10.233.78.216:2379 + set +x + tso='457114268232843266 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114268232843266 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:30 CST 2025] <<<<<< START cdc server in cli_tls_with_auth case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.4116841170.out server --log-file /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_cli_tls_with_auth_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli_tls_with_auth/cdc_data_cli_tls_with_auth_tls1 --cluster-id default --config /tmp/tidb_cdc_test/cli_tls_with_auth/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_file_large_value.check1 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:42:32.401 +08:00] [INFO] [main.go:86] ["running ddl test: 4 modifyColumnDDL"] + set +x failed proxying s2c rpc error: code = Canceled desc = context canceled + tso='457114268701032449 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114268701032449 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114268701032449 + echo 'query start ts: 457114268701032449' query start ts: 457114268701032449 + run_cdc_server --workdir /tmp/tidb_cdc_test/http_proxies --binary cdc.test [Fri Apr 4 15:42:32 CST 2025] <<<<<< START cdc server in http_proxies case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.1305413056.out server --log-file /tmp/tidb_cdc_test/http_proxies/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_proxies/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Uses proxy env variable http_proxy == 'http://127.0.0.1:13258' * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 13258 (#0) * Server auth using Basic with user 'ticdc' > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 502 Bad Gateway < Date: Fri, 04 Apr 2025 07:42:32 GMT < Content-Length: 0 < * Connection #0 to host 127.0.0.1 left intact + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:42:32.608 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:42:32.621 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled [2025/04/04 15:42:32.802 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:42:32.803 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:42:32.823 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:42:32.824 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:42:32.996 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:42:33.016 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:42:33.175 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:42:33.177 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:42:33.196 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:42:33.197 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] table consistent_replicate_storage_file_large_value.check1 exists table consistent_replicate_storage_file_large_value.t2 not exists for 1-th check, retry later [2025/04/04 15:42:33.360 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:42:33.379 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:33 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d18bd5b-81f7-405f-b75f-cd337241f792 {"id":"4d18bd5b-81f7-405f-b75f-cd337241f792","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752550} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc025edd5 4d18bd5b-81f7-405f-b75f-cd337241f792 /tidb/cdc/default/default/upstream/7489360132260082184 {"id":7489360132260082184,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d18bd5b-81f7-405f-b75f-cd337241f792 {"id":"4d18bd5b-81f7-405f-b75f-cd337241f792","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752550} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc025edd5 4d18bd5b-81f7-405f-b75f-cd337241f792 /tidb/cdc/default/default/upstream/7489360132260082184 {"id":7489360132260082184,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d18bd5b-81f7-405f-b75f-cd337241f792 {"id":"4d18bd5b-81f7-405f-b75f-cd337241f792","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752550} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc025edd5 4d18bd5b-81f7-405f-b75f-cd337241f792 /tidb/cdc/default/default/upstream/7489360132260082184 {"id":7489360132260082184,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 75e7d2aa-cb1a-4cc1-85ec-cda59ad5ae7b Info: {"upstream_id":7489360132260082184,"namespace":"default","id":"75e7d2aa-cb1a-4cc1-85ec-cda59ad5ae7b","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:42:33.310200023+08:00","start_ts":457114269315497990,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114269315497990,"checkpoint_ts":457114269315497990,"checkpoint_time":"2025-04-04 15:42:33.236"} [2025/04/04 15:42:33.542 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:42:33.544 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:42:33.554 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:42:33.555 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:42:33.715 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:42:33.729 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:42:33.893 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:42:33.895 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:42:33.909 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:42:33.910 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --user ticdc:ticdc_secret -vsL --max-time 20 https://127.0.0.1:8300/debug/info * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [6 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Request CERT (13): { [142 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [928 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [36 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Certificate (11): } [1814 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, CERT verify (15): } [264 bytes data] * TLSv1.3 (OUT), TLS handshake, [no content] (0): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [36 bytes data] * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * subjectAltName: host "127.0.0.1" matched cert's IP address! * issuer: C=CN; ST=Beijing; L=Beijing; O=PingCAP; CN=My own CA * SSL certificate verify ok. * Server auth using Basic with user 'ticdc' } [5 bytes data] * TLSv1.3 (OUT), TLS app data, [no content] (0): } [1 bytes data] > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, [no content] (0): { [1 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [2834 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:33 GMT < Content-Length: 1210 < Content-Type: text/plain; charset=utf-8 < { [1067 bytes data] * TLSv1.3 (IN), TLS app data, [no content] (0): { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a402912f-0a65-45c5-b234-99c85bf98aba {"id":"a402912f-0a65-45c5-b234-99c85bf98aba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752551} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc0330ff1 a402912f-0a65-45c5-b234-99c85bf98aba /tidb/cdc/default/default/upstream/7489360145989388168 {"id":7489360145989388168,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a402912f-0a65-45c5-b234-99c85bf98aba {"id":"a402912f-0a65-45c5-b234-99c85bf98aba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752551} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc0330ff1 a402912f-0a65-45c5-b234-99c85bf98aba /tidb/cdc/default/default/upstream/7489360145989388168 {"id":7489360145989388168,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a402912f-0a65-45c5-b234-99c85bf98aba {"id":"a402912f-0a65-45c5-b234-99c85bf98aba","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752551} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d95ffc0330ff1 a402912f-0a65-45c5-b234-99c85bf98aba /tidb/cdc/default/default/upstream/7489360145989388168 {"id":7489360145989388168,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x [2025/04/04 15:42:34.071 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:42:34.086 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] table autorandom_test.table_a not exists for 1-th check, retry later [2025/04/04 15:42:34.286 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:42:34.288 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:42:34.314 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:42:34.315 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:42:34.459 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:42:34.498 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:42:34.634 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:42:34.636 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:42:34.707 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:42:34.708 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360161958180943 TiDB Cluster ID. tikv_gc_leader_uuid 657ff017ec0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:15245, start at 2025-04-04 15:42:32.388317322 +0800 CST m=+2.244832342 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:32.395 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:32.426 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:32.426 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360161958180943 TiDB Cluster ID. tikv_gc_leader_uuid 657ff017ec0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:15245, start at 2025-04-04 15:42:32.388317322 +0800 CST m=+2.244832342 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:32.395 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:32.426 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:32.426 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360154731533899 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0181ec0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:15323, start at 2025-04-04 15:42:32.578650846 +0800 CST m=+2.337599381 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:32.586 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:32.571 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:32.571 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:42:34.829 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:42:34.894 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:42:35.015 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:42:35.017 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:42:35.082 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:42:35.083 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:42:35.195 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:42:35.268 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:42:35.376 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:42:35.378 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:42:35.459 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:42:35.460 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:42:35.560 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:42:35.642 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:42:35.741 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:42:35.743 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41224.out cli changefeed create --start-ts=457114268232843266 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name table consistent_replicate_storage_file_large_value.t2 not exists for 2-th check, retry later [2025/04/04 15:42:35.837 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:42:35.838 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:42:35.931 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Uses proxy env variable http_proxy == 'http://127.0.0.1:13258' * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 13258 (#0) * Server auth using Basic with user 'ticdc' > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 OK < Content-Length: 814 < Content-Type: text/plain; charset=utf-8 < Date: Fri, 04 Apr 2025 07:42:35 GMT < { [814 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fa96090c-c97d-4f2c-b4f1-ad3e46224a45 {"id":"fa96090c-c97d-4f2c-b4f1-ad3e46224a45","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752552} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/237b95ffbfc95e79 fa96090c-c97d-4f2c-b4f1-ad3e46224a45 /tidb/cdc/default/default/upstream/7489360025605239323 {"id":7489360025605239323,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.78.216:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fa96090c-c97d-4f2c-b4f1-ad3e46224a45 {"id":"fa96090c-c97d-4f2c-b4f1-ad3e46224a45","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752552} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/237b95ffbfc95e79 fa96090c-c97d-4f2c-b4f1-ad3e46224a45 /tidb/cdc/default/default/upstream/7489360025605239323 {"id":7489360025605239323,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.78.216:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fa96090c-c97d-4f2c-b4f1-ad3e46224a45 {"id":"fa96090c-c97d-4f2c-b4f1-ad3e46224a45","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752552} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/237b95ffbfc95e79 fa96090c-c97d-4f2c-b4f1-ad3e46224a45 /tidb/cdc/default/default/upstream/7489360025605239323 {"id":7489360025605239323,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.78.216:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + echo started cdc server successfully started cdc server successfully + SINK_URI=blackhole:/// + run_cdc_cli changefeed create --start-ts=457114268701032449 --sink-uri=blackhole:/// + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.13114.out cli changefeed create --start-ts=457114268701032449 --sink-uri=blackhole:/// Create changefeed successfully! ID: 913ebe2e-b3bd-443b-9a34-39ef152b219a Info: {"upstream_id":7489360025605239323,"namespace":"default","id":"913ebe2e-b3bd-443b-9a34-39ef152b219a","sink_uri":"blackhole:///","create_time":"2025-04-04T15:42:35.827393082+08:00","start_ts":457114268701032449,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114268701032449,"checkpoint_ts":457114268701032449,"checkpoint_time":"2025-04-04 15:42:30.892"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [WARN] --tz is deprecated in changefeed settings. [2025/04/04 15:42:36.027 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:42:36.110 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:42:36.112 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:42:36.211 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:42:36.212 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360145989388168,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:42:36.271474445+08:00","start_ts":457114268232843266,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114268232843266,"checkpoint_ts":457114268232843266,"checkpoint_time":"2025-04-04 15:42:29.106"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:42:36.294 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:42:36.402 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:42:36.478 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:42:36.479 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.16625.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:42:36.586 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:42:36.587 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:42:36.658 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:42:36.779 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] table autorandom_test.table_a exists check diff failed 1-th time, retry later [2025/04/04 15:42:36.848 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] [2025/04/04 15:42:36.850 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] [2025/04/04 15:42:36.958 +08:00] [INFO] [main.go:235] ["0 delete success: 1200"] [2025/04/04 15:42:36.959 +08:00] [INFO] [main.go:221] ["0 insert success: 2400"] [2025/04/04 15:42:37.030 +08:00] [INFO] [main.go:221] ["1 insert success: 2500"] [2025/04/04 15:42:37.137 +08:00] [INFO] [main.go:221] ["0 insert success: 2500"] [2025/04/04 15:42:37.211 +08:00] [INFO] [main.go:221] ["1 insert success: 2600"] [2025/04/04 15:42:37.213 +08:00] [INFO] [main.go:235] ["1 delete success: 1300"] + set +x + sleep 5 [2025/04/04 15:42:37.322 +08:00] [INFO] [main.go:235] ["0 delete success: 1300"] [2025/04/04 15:42:37.323 +08:00] [INFO] [main.go:221] ["0 insert success: 2600"] [2025/04/04 15:42:37.392 +08:00] [INFO] [main.go:221] ["1 insert success: 2700"] + set +x table test.simple not exists for 1-th check, retry later table consistent_replicate_storage_file_large_value.t2 not exists for 3-th check, retry later + set +x + tso='457114270280712193 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114270280712193 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:42:38 CST 2025] <<<<<< START cdc server in server_config_compatibility case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.1667116673.out server --log-file /tmp/tidb_cdc_test/server_config_compatibility/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/server_config_compatibility/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/server_config_compatibility/conf/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release wait process cdc.test exit for 1-th time... table consistent_replicate_storage_file_large_value.t2 not exists for 4-th check, retry later wait process cdc.test exit for 2-th time... + stop_tidb_cluster table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:40 CST 2025] <<<<<< run test case autorandom success! >>>>>> Starting Upstream TiDB... \033[0;36m<<< Run all test success >>>\033[0m table consistent_replicate_storage_file_large_value.t2 not exists for 5-th check, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:41 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf {"id":"d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752558} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc042f4f3 d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf /tidb/cdc/default/default/upstream/7489360161958180943 {"id":7489360161958180943,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf {"id":"d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752558} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc042f4f3 d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf /tidb/cdc/default/default/upstream/7489360161958180943 {"id":7489360161958180943,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf {"id":"d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752558} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc042f4f3 d7273c16-ab1b-4d21-aff8-aa5adb9d0aaf /tidb/cdc/default/default/upstream/7489360161958180943 {"id":7489360161958180943,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.16731.out cli changefeed create --start-ts=457114270280712193 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } Create changefeed successfully! ID: 010ed923-7616-4706-ace4-0250f5197bf7 Info: {"upstream_id":7489360161958180943,"namespace":"default","id":"010ed923-7616-4706-ace4-0250f5197bf7","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:42:41.980645597+08:00","start_ts":457114270280712193,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114270280712193,"checkpoint_ts":457114270280712193,"checkpoint_time":"2025-04-04 15:42:36.918"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // timeout [Pipeline] } table test.`simple-dash` exists + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114268626059284, "checkpoint_time": "2025-04-04 15:42:30.606", "error": null }' + echo '{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114268626059284, "checkpoint_time": "2025-04-04 15:42:30.606", "error": null }' { "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114268626059284, "checkpoint_time": "2025-04-04 15:42:30.606", "error": null } ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114268626059284, '"checkpoint_time":' '"2025-04-04' '15:42:30.606",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114268626059284, '"checkpoint_time":' '"2025-04-04' '15:42:30.606",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: https://127.0.0.1:2579 [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } + check + services=($(cat $WORK_DIR/test_proxy.log | xargs -L1 dirname | sort | uniq)) ++ cat /tmp/tidb_cdc_test/http_proxies/test_proxy.log ++ xargs -L1 dirname ++ sort ++ uniq + service_type_count=46 + echo 'captured services: ' captured services: + echo . /cdcpb.ChangeData /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager /tikvpb.Tikv '[2025/04' '[metadata=MD{:authority=[10.233.78.216:20162],' 'user-agent=[grpc-go/1.64.0],' 'grpc-accept-encoding=[gzip],' 'features=[stream-multiplexing],' 'content-type=[application' '[metadata=MD{:authority=[10.233.78.216:2379],' 'user-agent=[grpc-go/1.64.0],' 'grpc-accept-encoding=[gzip],' 'content-type=[application' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.78.216:20161],' 'features=[stream-multiplexing],' 'user-agent=[grpc-go' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.78.216:20161],' 'user-agent=[grpc-go' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.78.216:2379],' 'grpc-accept-encoding=[gzip],' 'user-agent=[grpc-go' '[metadata=MD{content-type=[application/grpc],' ':authority=[10.233.78.216:2379],' 'user-agent=[grpc-go' '[metadata=MD{grpc-accept-encoding=[gzip],' 'content-type=[application/grpc],' ':authority=[10.233.78.216:2379],' 'user-agent=[grpc-go' '[metadata=MD{grpc-accept-encoding=[gzip],' 'features=[stream-multiplexing],' 'content-type=[application/grpc],' ':authority=[10.233.78.216:20162],' 'user-agent=[grpc-go' '[metadata=MD{user-agent=[grpc-go/1.64.0],' 'grpc-accept-encoding=[gzip],' 'content-type=[application' '[method=/cdcpb.ChangeData' '[method=/pdpb.PD' '[method=/resource_manager.ResourceManager' '[stack=main.intercept\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go:56\ngithub.com/bradleyjkemp/grpc-tools/grpc-proxy.recoverWrapper.func1\n\t/go/pkg/mod/github.com/bradleyjkemp/grpc-tools@v0.2.5/grpc-proxy/config.go:52\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1680\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1800\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0' . /cdcpb.ChangeData /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager /tikvpb.Tikv [2025/04 [metadata=MD{:authority=[10.233.78.216:20162], user-agent=[grpc-go/1.64.0], grpc-accept-encoding=[gzip], features=[stream-multiplexing], content-type=[application [metadata=MD{:authority=[10.233.78.216:2379], user-agent=[grpc-go/1.64.0], grpc-accept-encoding=[gzip], content-type=[application [metadata=MD{content-type=[application/grpc], :authority=[10.233.78.216:20161], features=[stream-multiplexing], user-agent=[grpc-go [metadata=MD{content-type=[application/grpc], :authority=[10.233.78.216:20161], user-agent=[grpc-go [metadata=MD{content-type=[application/grpc], :authority=[10.233.78.216:2379], grpc-accept-encoding=[gzip], user-agent=[grpc-go [metadata=MD{content-type=[application/grpc], :authority=[10.233.78.216:2379], user-agent=[grpc-go [metadata=MD{grpc-accept-encoding=[gzip], content-type=[application/grpc], :authority=[10.233.78.216:2379], user-agent=[grpc-go [metadata=MD{grpc-accept-encoding=[gzip], features=[stream-multiplexing], content-type=[application/grpc], :authority=[10.233.78.216:20162], user-agent=[grpc-go [metadata=MD{user-agent=[grpc-go/1.64.0], grpc-accept-encoding=[gzip], content-type=[application [method=/cdcpb.ChangeData [method=/pdpb.PD [method=/resource_manager.ResourceManager [stack=main.intercept\n\t/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go:56\ngithub.com/bradleyjkemp/grpc-tools/grpc-proxy.recoverWrapper.func1\n\t/go/pkg/mod/github.com/bradleyjkemp/grpc-tools@v0.2.5/grpc-proxy/config.go:52\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1680\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1800\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.64.0 + '[' 46 -ge 2 ']' + check_logs /tmp/tidb_cdc_test/http_proxies ++ date + echo '[Fri Apr 4 15:42:42 CST 2025] <<<<<< run test case http_proxies success! >>>>>>' [Fri Apr 4 15:42:42 CST 2025] <<<<<< run test case http_proxies success! >>>>>> + stop_tidb_cluster failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled [Pipeline] // podTemplate [Pipeline] } [2025/04/04 15:42:42.510 +08:00] [INFO] [main.go:86] ["running ddl test: 5 addDropIndexDDL"] [2025/04/04 15:42:42.708 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:42:42.709 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [2025/04/04 15:42:42.931 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:42:42.932 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:42:42.937 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:42:42.939 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41397.out cli changefeed --changefeed-id custom-changefeed-name pause [2025/04/04 15:42:43.166 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:42:43.181 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] + set +x table consistent_replicate_storage_file_large_value.t2 not exists for 6-th check, retry later [2025/04/04 15:42:43.387 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:42:43.389 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:42:43.416 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:42:43.418 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:42:43.609 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:42:43.653 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore [2025/04/04 15:42:43.831 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:42:43.833 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:42:43.927 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:42:43.930 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] Starting Downstream TiDB... [2025/04/04 15:42:44.095 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:42:44.354 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:42:44.355 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:42:44.375 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:42:44.625 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:42:44.824 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:42:44.827 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] + set +x [2025/04/04 15:42:44.906 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:42:44.907 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:42:45.224 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:42:45.244 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:42:45.559 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:42:45.561 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:42:45.644 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:42:45.649 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.t2 exists check diff successfully [2025/04/04 15:42:45.890 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:42:46.058 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later [2025/04/04 15:42:46.222 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:42:46.223 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] wait process cdc.test exit for 1-th time... [2025/04/04 15:42:46.479 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:42:46.483 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:42:46.599 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] wait process cdc.test exit for 2-th time... [2025/04/04 15:42:46.949 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:42:47.012 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:42:47.013 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:47 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1430214304.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:42:47.365 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:42:47.383 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:42:47.386 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360208049510673 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0266180038 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:15662, start at 2025-04-04 15:42:47.191750592 +0800 CST m=+3.435607869 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:47.199 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:47.174 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:47.174 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360208049510673 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0266180038 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:15662, start at 2025-04-04 15:42:47.191750592 +0800 CST m=+3.435607869 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:47.199 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:47.174 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:47.174 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360208110743908 TiDB Cluster ID. tikv_gc_leader_uuid 657ff026920000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:15721, start at 2025-04-04 15:42:47.385740593 +0800 CST m=+1.726811925 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:44:47.392 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:42:47.368 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:32:47.368 +0800 All versions after safe point can be accessed. (DO NOT EDIT) check data successfully Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114271771787271, "checkpoint_time": "2025-04-04 15:42:42.606", "error": null }' + echo '{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114271771787271, "checkpoint_time": "2025-04-04 15:42:42.606", "error": null }' { "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 457114271771787271, "checkpoint_time": "2025-04-04 15:42:42.606", "error": null } ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114271771787271, '"checkpoint_time":' '"2025-04-04' '15:42:42.606",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 457114271771787271, '"checkpoint_time":' '"2025-04-04' '15:42:42.606",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41475.out cli changefeed update --pd=https://127.0.0.1:2579 --config=/tmp/tidb_cdc_test/cli_tls_with_auth/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} wait process cdc.test exit for 1-th time... Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360145989388168,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:42:36.271474445+08:00","start_ts":457114268232843266,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"content_compatible":null,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":0,"checkpoint_ts":457114271771787271,"checkpoint_time":"2025-04-04 15:42:42.606"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:42:49 CST 2025] <<<<<< run test case server_config_compatibility success! >>>>>> + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41505.out cli changefeed --changefeed-id custom-changefeed-name resume + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.16975.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:50 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/8d517938-3586-4734-b970-ddc44d3b4888 {UpstreamID:7489360069113822555 Namespace:default ID:8d517938-3586-4734-b970-ddc44d3b4888 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:19.624989902 +0800 CST StartTs:457114265714950148 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004142900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114265741164549} {CheckpointTs:457114268283961373 MinTableBarrierTs:457114273697234952 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/da895098-0032-4ab5-bce5-ab29d8f5e021 {"id":"da895098-0032-4ab5-bce5-ab29d8f5e021","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752567} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbff39145 da895098-0032-4ab5-bce5-ab29d8f5e021 /tidb/cdc/default/default/changefeed/info/8d517938-3586-4734-b970-ddc44d3b4888 {"upstream-id":7489360069113822555,"namespace":"default","changefeed-id":"8d517938-3586-4734-b970-ddc44d3b4888","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:42:19.624989902+08:00","start-ts":457114265714950148,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114265741164549} /tidb/cdc/default/default/changefeed/status/8d517938-3586-4734-b970-ddc44d3b4888 {"checkpoint-ts":457114268283961373,"min-table-barrier-ts":457114273697234952,"admin-job-type":0} /tidb/cdc/default/default/task/position/da895098-0032-4ab5-bce5-ab29d8f5e021/8d517938-3586-4734-b970-ddc44d3b4888 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360069113822555 {"id":7489360069113822555,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d517938-3586-4734-b970-ddc44d3b4888 {UpstreamID:7489360069113822555 Namespace:default ID:8d517938-3586-4734-b970-ddc44d3b4888 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:19.624989902 +0800 CST StartTs:457114265714950148 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004142900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114265741164549} {CheckpointTs:457114268283961373 MinTableBarrierTs:457114273697234952 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/da895098-0032-4ab5-bce5-ab29d8f5e021 {"id":"da895098-0032-4ab5-bce5-ab29d8f5e021","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752567} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbff39145 da895098-0032-4ab5-bce5-ab29d8f5e021 /tidb/cdc/default/default/changefeed/info/8d517938-3586-4734-b970-ddc44d3b4888 {"upstream-id":7489360069113822555,"namespace":"default","changefeed-id":"8d517938-3586-4734-b970-ddc44d3b4888","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:42:19.624989902+08:00","start-ts":457114265714950148,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114265741164549} /tidb/cdc/default/default/changefeed/status/8d517938-3586-4734-b970-ddc44d3b4888 {"checkpoint-ts":457114268283961373,"min-table-barrier-ts":457114273697234952,"admin-job-type":0} /tidb/cdc/default/default/task/position/da895098-0032-4ab5-bce5-ab29d8f5e021/8d517938-3586-4734-b970-ddc44d3b4888 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360069113822555 {"id":7489360069113822555,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d517938-3586-4734-b970-ddc44d3b4888 {UpstreamID:7489360069113822555 Namespace:default ID:8d517938-3586-4734-b970-ddc44d3b4888 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:42:19.624989902 +0800 CST StartTs:457114265714950148 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc004142900 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114265741164549} {CheckpointTs:457114268283961373 MinTableBarrierTs:457114273697234952 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114268283961373, checkpointTs: 457114268283961373, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/da895098-0032-4ab5-bce5-ab29d8f5e021 {"id":"da895098-0032-4ab5-bce5-ab29d8f5e021","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752567} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffbff39145 da895098-0032-4ab5-bce5-ab29d8f5e021 /tidb/cdc/default/default/changefeed/info/8d517938-3586-4734-b970-ddc44d3b4888 {"upstream-id":7489360069113822555,"namespace":"default","changefeed-id":"8d517938-3586-4734-b970-ddc44d3b4888","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:42:19.624989902+08:00","start-ts":457114265714950148,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114265741164549} /tidb/cdc/default/default/changefeed/status/8d517938-3586-4734-b970-ddc44d3b4888 {"checkpoint-ts":457114268283961373,"min-table-barrier-ts":457114273697234952,"admin-job-type":0} /tidb/cdc/default/default/task/position/da895098-0032-4ab5-bce5-ab29d8f5e021/8d517938-3586-4734-b970-ddc44d3b4888 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360069113822555 {"id":7489360069113822555,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... + set +x + set +x + tso='457114274044051457 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114274044051457 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114274044051457 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:42:52 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1700517007.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/generate_column Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114274930622466, "checkpoint_time": "2025-04-04 15:42:54.656", "error": null }' + echo '{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114274930622466, "checkpoint_time": "2025-04-04 15:42:54.656", "error": null }' { "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114274930622466, "checkpoint_time": "2025-04-04 15:42:54.656", "error": null } ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114274930622466, '"checkpoint_time":' '"2025-04-04' '15:42:54.656",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114274930622466, '"checkpoint_time":' '"2025-04-04' '15:42:54.656",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41582.out cli changefeed --changefeed-id custom-changefeed-name remove Verifying downstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:42:55 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39 {"id":"ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752573} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc06b67f5 ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39 /tidb/cdc/default/default/upstream/7489360208049510673 {"id":7489360208049510673,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39 {"id":"ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752573} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc06b67f5 ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39 /tidb/cdc/default/default/upstream/7489360208049510673 {"id":7489360208049510673,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39 {"id":"ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752573} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc06b67f5 ff9f6c3b-f5f8-46d7-b352-a7d7a3059d39 /tidb/cdc/default/default/upstream/7489360208049510673 {"id":7489360208049510673,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114274044051457 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.17058.out cli changefeed create --start-ts=457114274044051457 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360208049510673,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:42:56.254089412+08:00","start_ts":457114274044051457,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114274044051457,"checkpoint_ts":457114274044051457,"checkpoint_time":"2025-04-04 15:42:51.274"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 457114275192766466 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/ PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:42:56.917 +08:00] [INFO] [main.go:78] ["runDDLTest take 1m5.021200689s"] table mark.finish_mark_0 exists table mark.finish_mark_1 not exists for 1-th check, retry later + stop_proxy + kill 8996 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sequence/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 10045 0 --:--:-- --:--:-- --:--:-- 10045 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:42:51.274","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:42:57.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:42:51.274","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:42:57.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:42:51.274","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:42:57.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2025-04-04 15:42:51.274' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:42:51.274","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:42:57.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:42:51.274","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:42:57.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2025-04-04 15:42:57' + echo 'sink_checkpoint_ts is 2025-04-04' 15:42:51.274 sink_checkpoint_ts is 2025-04-04 15:42:51.274 ++ date -d '2025-04-04 15:42:51.274' +%s + checkpoint_timestamp=1743752571 ++ date -d '2025-04-04 15:42:57' +%s + current_timestamp=1743752577 + '[' 6 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... + check_table_exists test.t1 127.0.0.1 3306 table test.t1 exists + sleep 5 Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ci_collation_compatibility.t exists table ci_collation_compatibility.finish_mark_1 exists check diff successfully wait process cdc.test exit for 1-th time... table mark.finish_mark_1 not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sequence Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 4 15:43:00 CST 2025] <<<<<< run test case ci_collation_compatibility success! >>>>>> changefeed count 0 check pass, pd_addr: https://127.0.0.1:2579 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41640.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/changefeed_dup_error_restart/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7489360145989388168,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:01.518309127+08:00","start_ts":457114276700094468,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114276700094468,"checkpoint_ts":457114276700094468,"checkpoint_time":"2025-04-04 15:43:01.406"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table mark.finish_mark_1 not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360282470717121 TiDB Cluster ID. tikv_gc_leader_uuid 657ff033a340008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:30842, start at 2025-04-04 15:43:00.765536196 +0800 CST m=+2.493734298 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:00.773 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:00.749 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:00.749 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360282470717121 TiDB Cluster ID. tikv_gc_leader_uuid 657ff033a340008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:30842, start at 2025-04-04 15:43:00.765536196 +0800 CST m=+2.493734298 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:00.773 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:00.749 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:00.749 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360278625920045 TiDB Cluster ID. tikv_gc_leader_uuid 657ff033d4c000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-8bgqv-08rx2-kgk1j, pid:30923, start at 2025-04-04 15:43:00.96751018 +0800 CST m=+2.602409063 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:00.975 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:00.947 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:00.947 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/generate_column/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/generate_column/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x table mark.finish_mark_1 not exists for 4-th check, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 48600 0 --:--:-- --:--:-- --:--:-- 48600 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:43:02.075","puller_resolved_ts":"2025-04-04 15:42:57.774","last_synced_ts":"2025-04-04 15:42:58.275","now_ts":"2025-04-04 15:43:03.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:43:02.075","puller_resolved_ts":"2025-04-04' '15:42:57.774","last_synced_ts":"2025-04-04' '15:42:58.275","now_ts":"2025-04-04' '15:43:03.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:43:02.075","puller_resolved_ts":"2025-04-04' '15:42:57.774","last_synced_ts":"2025-04-04' '15:42:58.275","now_ts":"2025-04-04' '15:43:03.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.32298.out cli tso query --pd=http://127.0.0.1:2379 start tidb cluster in /tmp/tidb_cdc_test/changefeed_dup_error_restart Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_1 not exists for 5-th check, retry later + endpoints=https://127.0.0.1:2579 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c custom-changefeed-name -s + info='{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114277814468612, "checkpoint_time": "2025-04-04 15:43:05.657", "error": null }' + echo '{ "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114277814468612, "checkpoint_time": "2025-04-04 15:43:05.657", "error": null }' { "upstream_id": 7489360145989388168, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 457114277814468612, "checkpoint_time": "2025-04-04 15:43:05.657", "error": null } ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114277814468612, '"checkpoint_time":' '"2025-04-04' '15:43:05.657",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360145989388168, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 457114277814468612, '"checkpoint_time":' '"2025-04-04' '15:43:05.657",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + set +x + tso='457114277655085058 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114277655085058 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:43:06 CST 2025] <<<<<< START cdc server in generate_column case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.3232932331.out server --log-file /tmp/tidb_cdc_test/generate_column/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/generate_column/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41742.out cli unsafe delete-service-gc-safepoint table mark.finish_mark_1 not exists for 6-th check, retry later Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41774.out cli unsafe reset --no-confirm --pd=https://127.0.0.1:2579 Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360308862932235 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039798000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:14163, start at 2025-04-04 15:43:06.745504543 +0800 CST m=+2.149675015 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.753 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.726 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.726 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360308862932235 TiDB Cluster ID. tikv_gc_leader_uuid 657ff039798000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:14163, start at 2025-04-04 15:43:06.745504543 +0800 CST m=+2.149675015 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.753 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.726 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.726 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360315091544942 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0398640002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-smdjj-qrvfs-d7xt8, pid:14228, start at 2025-04-04 15:43:06.77885246 +0800 CST m=+2.109994854 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:06.827 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:06.827 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:06.827 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/sequence/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/sequence/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sequence/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } reset and all metadata truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:09 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2316a255-14ed-41de-af6b-64f20b4a297e {"id":"2316a255-14ed-41de-af6b-64f20b4a297e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752586} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0b0d5e3 2316a255-14ed-41de-af6b-64f20b4a297e /tidb/cdc/default/default/upstream/7489360282470717121 {"id":7489360282470717121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2316a255-14ed-41de-af6b-64f20b4a297e {"id":"2316a255-14ed-41de-af6b-64f20b4a297e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752586} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0b0d5e3 2316a255-14ed-41de-af6b-64f20b4a297e /tidb/cdc/default/default/upstream/7489360282470717121 {"id":7489360282470717121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2316a255-14ed-41de-af6b-64f20b4a297e {"id":"2316a255-14ed-41de-af6b-64f20b4a297e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752586} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0b0d5e3 2316a255-14ed-41de-af6b-64f20b4a297e /tidb/cdc/default/default/upstream/7489360282470717121 {"id":7489360282470717121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.32385.out cli changefeed create --start-ts=457114277655085058 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ table mark.finish_mark_1 not exists for 7-th check, retry later Create changefeed successfully! ID: 6d6a0cee-bc76-4901-be4d-c94e264b6633 Info: {"upstream_id":7489360282470717121,"namespace":"default","id":"6d6a0cee-bc76-4901-be4d-c94e264b6633","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:09.994175304+08:00","start_ts":457114277655085058,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114277655085058,"checkpoint_ts":457114277655085058,"checkpoint_time":"2025-04-04 15:43:05.049"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } + set +x Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Fri Apr 4 15:43:11 CST 2025] <<<<<< START cdc server in sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sequence.1563915641.out server --log-file /tmp/tidb_cdc_test/sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sequence/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x table generate_column.t1 not exists for 1-th check, retry later table mark.finish_mark_1 not exists for 8-th check, retry later check_redo_resolved_ts 8d517938-3586-4734-b970-ddc44d3b4888 457114279215366153 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/8d517938-3586-4734-b970-ddc44d3b4888/meta global resolved ts 457114278691078147 not forward to 457114279215366153 run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_redo_resolved_ts 8d517938-3586-4734-b970-ddc44d3b4888 457114279215366153 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/8d517938-3586-4734-b970-ddc44d3b4888/meta global resolved ts 457114279215366147 not forward to 457114279215366153 run task failed 2-th time, retry later table mark.finish_mark_1 not exists for 9-th check, retry later table generate_column.t1 exists table generate_column.t2 not exists for 1-th check, retry later + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41862.out cli unsafe resolve-lock --region=60 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:14 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5 {"id":"ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0cb82d7 ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5 /tidb/cdc/default/default/upstream/7489360308862932235 {"id":7489360308862932235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5 {"id":"ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0cb82d7 ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5 /tidb/cdc/default/default/upstream/7489360308862932235 {"id":7489360308862932235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5 {"id":"ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752591} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0cb82d7 ea3a4b7a-de2c-44c1-871a-1bbe5915fbd5 /tidb/cdc/default/default/upstream/7489360308862932235 {"id":7489360308862932235,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sequence.cli.15699.out cli changefeed create --start-ts=457114279221919745 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/sequence/conf/force_replicate.toml Create changefeed successfully! ID: 869d3ee9-b0cf-4225-9a1f-a3f7cd1e7592 Info: {"upstream_id":7489360308862932235,"namespace":"default","id":"869d3ee9-b0cf-4225-9a1f-a3f7cd1e7592","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:43:14.610414886+08:00","start_ts":457114279221919745,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114279221919745,"checkpoint_ts":457114279221919745,"checkpoint_time":"2025-04-04 15:43:11.026"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360329268610086 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03f8ac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:17770, start at 2025-04-04 15:43:12.975737865 +0800 CST m=+2.180283769 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:12.983 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:12.989 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:12.989 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360329268610086 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03f8ac000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:17770, start at 2025-04-04 15:43:12.975737865 +0800 CST m=+2.180283769 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:12.983 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:12.989 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:12.989 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360337765442176 TiDB Cluster ID. tikv_gc_leader_uuid 657ff03fa30000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:17840, start at 2025-04-04 15:43:13.072291084 +0800 CST m=+2.201591711 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:13.080 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:13.086 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:13.086 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/changefeed_dup_error_restart/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_1 not exists for 10-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli_tls_with_auth.cli.41885.out cli unsafe resolve-lock --region=60 --ts=457114279151140870 table generate_column.t2 exists check diff failed 1-th time, retry later + set +x table sequence_test.t1 not exists for 1-th check, retry later PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... table mark.finish_mark_1 not exists for 11-th check, retry later + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 14133 400 --:--:-- --:--:-- --:--:-- 14533 { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" check_redo_resolved_ts 8d517938-3586-4734-b970-ddc44d3b4888 457114279215366153 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/8d517938-3586-4734-b970-ddc44d3b4888/meta run task successfully + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.cli.19217.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 1-th time... check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table sequence_test.t1 exists check table exists success check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [2025/04/04 15:43:18.959 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:43:18.960 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:43:18.960 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:43:18.960 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:43:18.961 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114273920057379] [resolvedTs=457114280264204290] [2025/04/04 15:43:18.961 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=842.773ยตs] [2025/04/04 15:43:18.961 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=933.744ยตs] [2025/04/04 15:43:18.961 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:43:18.961 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:43:18.962 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:43:18.967 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:43:18.969 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:43:18.971 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:43:18.971 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:43:18.971 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:43:18.971 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:43:18.975 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:43:18.977 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:43:18.977 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:43:18.978 +08:00] [ERROR] [redo.go:272] ["ignore unsupported DDL"] [ddl="{\"StartTs\":457114273920057373,\"CommitTs\":457114273920057379,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file_large_value`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file_large_value`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_storage_file_large_value\",\"Table\":\"t1\",\"TableID\":118,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] + set +x + tso='457114280929263618 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114280929263618 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "scanproportion"="0" "recordcount"="10" "insertproportion"="0" "mysql.db"="changefeed_dup_error_restart" "workload"="core" "readallfields"="true" "requestdistribution"="uniform" "dotransactions"="false" "mysql.host"="127.0.0.1" "updateproportion"="0" "readproportion"="0" "operationcount"="0" "threadcount"="2" "mysql.user"="root" "mysql.port"="4000" ********************************************** Run finished, takes 7.629346ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2137.9, Avg(us): 1470, Min(us): 912, Max(us): 2911, 95th(us): 3000, 99th(us): 3000 [Fri Apr 4 15:43:19 CST 2025] <<<<<< START cdc server in changefeed_dup_error_restart case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLDuplicateEntryError=5%return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.1927419276.out server --log-file /tmp/tidb_cdc_test/changefeed_dup_error_restart/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_dup_error_restart/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) [2025/04/04 15:43:19.179 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114280264204290] [2025/04/04 15:43:19.179 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114280264204290] ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:19 CST 2025] <<<<<< run test case generate_column success! >>>>>> [2025/04/04 15:43:19.480 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114280264204290] [2025/04/04 15:43:19.480 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114280264204290] [2025/04/04 15:43:19.580 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=8] [appliedDDLCount=0] [currentCheckpoint=457114280264204290] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:43:19.580 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] Apply redo log successfully table mark.finish_mark_1 not exists for 12-th check, retry later check diff successfully [Fri Apr 4 15:43:20 CST 2025] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> count(*) 0 check diff successfully ERROR 1146 (42S02) at line 1: Table 'sequence_test.seq0' doesn't exist table sequence_test.mark_table not exists for 1-th check, retry later } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 183 100 183 0 0 11437 0 --:--:-- --:--:-- --:--:-- 11437 { "version": "v9.0.0-beta.1-10-ga306f4894", "git_hash": "a306f48949e037965a7c1963f500f7e0abbfba51", "id": "8c1c531c-89dc-404c-9203-20ecbae497e9", "pid": 41173, "is_owner": true }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table mark.finish_mark_1 not exists for 13-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:22 CST 2025] <<<<<< run test case cli_tls_with_auth success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:22 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eccb6dbc-839d-45b2-9ce8-c154f6452d2a {"id":"eccb6dbc-839d-45b2-9ce8-c154f6452d2a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752599} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ddb8f7 eccb6dbc-839d-45b2-9ce8-c154f6452d2a /tidb/cdc/default/default/upstream/7489360329268610086 {"id":7489360329268610086,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eccb6dbc-839d-45b2-9ce8-c154f6452d2a {"id":"eccb6dbc-839d-45b2-9ce8-c154f6452d2a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752599} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ddb8f7 eccb6dbc-839d-45b2-9ce8-c154f6452d2a /tidb/cdc/default/default/upstream/7489360329268610086 {"id":7489360329268610086,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eccb6dbc-839d-45b2-9ce8-c154f6452d2a {"id":"eccb6dbc-839d-45b2-9ce8-c154f6452d2a","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752599} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc0ddb8f7 eccb6dbc-839d-45b2-9ce8-c154f6452d2a /tidb/cdc/default/default/upstream/7489360329268610086 {"id":7489360329268610086,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_dup_error_restart.cli.19330.out cli changefeed create --start-ts=457114280929263618 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' Create changefeed successfully! ID: 3b47345f-52ac-44fd-be1e-d3c38ed0fe26 Info: {"upstream_id":7489360329268610086,"namespace":"default","id":"3b47345f-52ac-44fd-be1e-d3c38ed0fe26","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:43:22.67974273+08:00","start_ts":457114280929263618,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114280929263618,"checkpoint_ts":457114280929263618,"checkpoint_time":"2025-04-04 15:43:17.539"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table sequence_test.mark_table exists wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table mark.finish_mark_1 not exists for 14-th check, retry later + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:24 CST 2025] <<<<<< run test case sequence success! >>>>>> table mark.finish_mark_1 not exists for 15-th check, retry later table mark.finish_mark_1 not exists for 16-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } table mark.finish_mark_1 not exists for 17-th check, retry later Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: mysql... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330DC2E8A7DD21 < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:43:31 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... table mark.finish_mark_1 not exists for 18-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/debezium_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:31 CST 2025] <<<<<< run test case debezium_basic success! >>>>>> table mark.finish_mark_1 not exists for 19-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:35 CST 2025] <<<<<< run test case kafka_simple_basic success! >>>>>> table mark.finish_mark_1 not exists for 20-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_s3 Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table mark.finish_mark_1 not exists for 21-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_1 not exists for 22-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_basic_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:38 CST 2025] <<<<<< run test case kafka_simple_basic_avro success! >>>>>> ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 24100 0 --:--:-- --:--:-- --:--:-- 23322 100 723 100 723 0 0 24100 0 --:--:-- --:--:-- --:--:-- 23322 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:41:17.523","puller_resolved_ts":"2025-04-04 15:41:17.523","last_synced_ts":"2025-04-04 15:41:10.623","now_ts":"2025-04-04 15:43:27.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:41:17.523","puller_resolved_ts":"2025-04-04' '15:41:17.523","last_synced_ts":"2025-04-04' '15:41:10.623","now_ts":"2025-04-04' '15:43:27.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:41:17.523","puller_resolved_ts":"2025-04-04' '15:41:17.523","last_synced_ts":"2025-04-04' '15:41:10.623","now_ts":"2025-04-04' '15:43:27.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_1 not exists for 23-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:42 CST 2025] <<<<<< run test case kafka_simple_handle_key_only success! >>>>>> chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360454619399913 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05b58c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:15631, start at 2025-04-04 15:43:41.460292995 +0800 CST m=+2.510908479 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.469 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.461 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.461 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360454619399913 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05b58c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:15631, start at 2025-04-04 15:43:41.460292995 +0800 CST m=+2.510908479 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.469 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.461 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.461 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360460889292727 TiDB Cluster ID. tikv_gc_leader_uuid 657ff05b65c000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:15702, start at 2025-04-04 15:43:41.479317433 +0800 CST m=+2.410526494 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:41.486 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:41.463 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:41.463 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_1 not exists for 24-th check, retry later [Fri Apr 4 15:43:45 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1700717009.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_1 not exists for 25-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_handle_key_only_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:45 CST 2025] <<<<<< run test case kafka_simple_handle_key_only_avro success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table mark.finish_mark_1 not exists for 26-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:48 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1dff33bc-c21b-4ec4-ba68-b69f7b400084 {"id":"1dff33bc-c21b-4ec4-ba68-b69f7b400084","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15588d9 1dff33bc-c21b-4ec4-ba68-b69f7b400084 /tidb/cdc/default/default/upstream/7489360454619399913 {"id":7489360454619399913,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1dff33bc-c21b-4ec4-ba68-b69f7b400084 {"id":"1dff33bc-c21b-4ec4-ba68-b69f7b400084","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15588d9 1dff33bc-c21b-4ec4-ba68-b69f7b400084 /tidb/cdc/default/default/upstream/7489360454619399913 {"id":7489360454619399913,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1dff33bc-c21b-4ec4-ba68-b69f7b400084 {"id":"1dff33bc-c21b-4ec4-ba68-b69f7b400084","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752625} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc15588d9 1dff33bc-c21b-4ec4-ba68-b69f7b400084 /tidb/cdc/default/default/upstream/7489360454619399913 {"id":7489360454619399913,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "mysql.user"="root" "insertproportion"="0" "readproportion"="0" "dotransactions"="false" "workload"="core" "fieldcount"="100" "updateproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="consistent_replicate_storage_s3" "threadcount"="10" "recordcount"="5000" "requestdistribution"="uniform" "mysql.port"="4000" "operationcount"="0" "readallfields"="true" "scanproportion"="0" ********************************************** Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:49 CST 2025] <<<<<< run test case kafka_simple_claim_check success! >>>>>> table mark.finish_mark_1 not exists for 27-th check, retry later Run finished, takes 2.003017535s INSERT - Takes(s): 2.0, Count: 5000, OPS: 2505.7, Avg(us): 3609, Min(us): 1238, Max(us): 59915, 95th(us): 4000, 99th(us): 51000 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.usertable not exists for 1-th check, retry later table mark.finish_mark_1 not exists for 28-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_simple_claim_check_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:52 CST 2025] <<<<<< run test case kafka_simple_claim_check_avro success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360497293563763 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06457c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:15162, start at 2025-04-04 15:43:50.643740936 +0800 CST m=+2.055377854 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:50.650 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:50.623 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:50.623 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360497293563763 TiDB Cluster ID. tikv_gc_leader_uuid 657ff06457c000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:15162, start at 2025-04-04 15:43:50.643740936 +0800 CST m=+2.055377854 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:50.650 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:50.623 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:50.623 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360491320135119 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0647c4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:15229, start at 2025-04-04 15:43:50.795631991 +0800 CST m=+2.139803647 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:45:50.806 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:43:50.823 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:33:50.823 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table consistent_replicate_storage_s3.usertable exists table consistent_replicate_storage_s3.check1 not exists for 1-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_1 exists table mark.finish_mark_2 not exists for 1-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 2-th check, retry later + cd /tmp/tidb_cdc_test/synced_status_with_redo ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16615.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:43:51.127 +08:00] [WARN] [diff.go:766] ["rows is not equal"] [table=`test`.`ntest`] [where="((TRUE) AND TRUE)"] [cost=1m32.16647249s] [2025/04/04 15:43:51.130 +08:00] [WARN] [diff.go:384] ["check chunk data not equal"] [chunk="{\"id\":0,\"bounds\":[],\"where\":\"((TRUE) AND TRUE)\",\"args\":null,\"state\":\"failed\"}"] table mark.finish_mark_2 not exists for 2-th check, retry later table changefeed_dup_error_restart.finish_mark_1 exists check diff successfully ***************** properties ***************** "operationcount"="0" "readproportion"="0" "scanproportion"="0" "mysql.db"="changefeed_dup_error_restart" "recordcount"="10" "insertproportion"="0" "readallfields"="true" "mysql.host"="127.0.0.1" "workload"="core" "requestdistribution"="uniform" "dotransactions"="false" "mysql.port"="4000" "threadcount"="2" "mysql.user"="root" "updateproportion"="0" ********************************************** Run finished, takes 4.065491ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3455.7, Avg(us): 717, Min(us): 437, Max(us): 1729, 95th(us): 2000, 99th(us): 2000 + set +x + tso='457114290820218881 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114290820218881 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114290820218881 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:43:56 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1664916651.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:56 CST 2025] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> [2025/04/04 15:43:56.697 +08:00] [INFO] [dailytest.go:68] ["test pass!!!"] table consistent_replicate_storage_s3.check1 not exists for 3-th check, retry later wait process cdc.test exit for 1-th time... table mark.finish_mark_2 not exists for 3-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:43:58 CST 2025] <<<<<< run test case cdc success! >>>>>> table consistent_replicate_storage_s3.check1 not exists for 4-th check, retry later table mark.finish_mark_2 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:43:59 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e8f7eb99-c170-4d62-9706-9c7a7ade3c92 {"id":"e8f7eb99-c170-4d62-9706-9c7a7ade3c92","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc17181e1 e8f7eb99-c170-4d62-9706-9c7a7ade3c92 /tidb/cdc/default/default/upstream/7489360497293563763 {"id":7489360497293563763,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e8f7eb99-c170-4d62-9706-9c7a7ade3c92 {"id":"e8f7eb99-c170-4d62-9706-9c7a7ade3c92","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc17181e1 e8f7eb99-c170-4d62-9706-9c7a7ade3c92 /tidb/cdc/default/default/upstream/7489360497293563763 {"id":7489360497293563763,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e8f7eb99-c170-4d62-9706-9c7a7ade3c92 {"id":"e8f7eb99-c170-4d62-9706-9c7a7ade3c92","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752637} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc17181e1 e8f7eb99-c170-4d62-9706-9c7a7ade3c92 /tidb/cdc/default/default/upstream/7489360497293563763 {"id":7489360497293563763,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114290820218881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.16715.out cli changefeed create --start-ts=457114290820218881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:43:59 CST 2025] <<<<<< run test case canal_json_basic success! >>>>>> Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489360497293563763,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:44:00.237390799+08:00","start_ts":457114290820218881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114290820218881,"checkpoint_ts":457114290820218881,"checkpoint_time":"2025-04-04 15:43:55.270"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 5-th check, retry later table mark.finish_mark_2 not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_content_compatible/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:02 CST 2025] <<<<<< run test case canal_json_content_compatible success! >>>>>> table consistent_replicate_storage_s3.check1 not exists for 6-th check, retry later table mark.finish_mark_2 not exists for 6-th check, retry later table test.t1 exists + sleep 5 table consistent_replicate_storage_s3.check1 not exists for 7-th check, retry later table mark.finish_mark_2 not exists for 7-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:06 CST 2025] <<<<<< run test case multi_topics success! >>>>>> table consistent_replicate_storage_s3.check1 not exists for 8-th check, retry later table mark.finish_mark_2 not exists for 8-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 9-th check, retry later + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo + info='jenkins 15162 9.4 0.0 3332324 195264 ? Sl 15:43 0:01 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743752628576545720.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb.log jenkins 15171 4.1 0.0 2666624 167040 ? Sl 15:43 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743752628585527700.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_other.log jenkins 15229 10.3 0.0 2815528 198720 ? Sl 15:43 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status_with_redo/tidb-config-1743752628644440974.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status_with_redo/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status_with_redo ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2700 0 --:--:-- --:--:-- --:--:-- 2670 100 243 100 243 0 0 2700 0 --:--:-- --:--:-- --:--:-- 2670 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:44:07.620","puller_resolved_ts":"2025-04-04 15:44:01.670","last_synced_ts":"2025-04-04 15:44:01.720","now_ts":"2025-04-04 15:44:08.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:44:07.620","puller_resolved_ts":"2025-04-04' '15:44:01.670","last_synced_ts":"2025-04-04' '15:44:01.720","now_ts":"2025-04-04' '15:44:08.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:44:07.620","puller_resolved_ts":"2025-04-04' '15:44:01.670","last_synced_ts":"2025-04-04' '15:44:01.720","now_ts":"2025-04-04' '15:44:08.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table mark.finish_mark_2 not exists for 9-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:09 CST 2025] <<<<<< run test case avro_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table consistent_replicate_storage_s3.check1 not exists for 10-th check, retry later table mark.finish_mark_2 exists table mark.finish_mark_3 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:13 CST 2025] <<<<<< run test case canal_json_handle_key_only success! >>>>>> table consistent_replicate_storage_s3.check1 not exists for 11-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... table mark.finish_mark_3 not exists for 2-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 12-th check, retry later table mark.finish_mark_3 not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:16 CST 2025] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> table consistent_replicate_storage_s3.check1 not exists for 13-th check, retry later table mark.finish_mark_3 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 5-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 14-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:20 CST 2025] <<<<<< run test case canal_json_claim_check success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 15-th check, retry later table mark.finish_mark_3 not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360627009166534 TiDB Cluster ID. tikv_gc_leader_uuid 657ff081758000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:25626, start at 2025-04-04 15:44:20.483925301 +0800 CST m=+2.123774900 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:20.534 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:20.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:20.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360627009166534 TiDB Cluster ID. tikv_gc_leader_uuid 657ff081758000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:25626, start at 2025-04-04 15:44:20.483925301 +0800 CST m=+2.123774900 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:20.534 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:20.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:20.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360629126083473 TiDB Cluster ID. tikv_gc_leader_uuid 657ff081994000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-v1kts-zfc7f-wc9c3, pid:25699, start at 2025-04-04 15:44:20.628992578 +0800 CST m=+2.182177826 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:20.637 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:20.630 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:20.630 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table consistent_replicate_storage_s3.check1 exists table consistent_replicate_storage_s3.t2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:23 CST 2025] <<<<<< run test case open_protocol_claim_check success! >>>>>> table mark.finish_mark_3 not exists for 7-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.27013.out cli tso query --pd=http://127.0.0.1:2379 table consistent_replicate_storage_s3.t2 not exists for 2-th check, retry later table mark.finish_mark_3 not exists for 8-th check, retry later + set +x + tso='457114298649935873 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114298649935873 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:44:26 CST 2025] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.2704227044.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:26 CST 2025] <<<<<< run test case canal_json_storage_basic success! >>>>>> table consistent_replicate_storage_s3.t2 not exists for 3-th check, retry later table mark.finish_mark_3 exists table mark.finish_mark_4 not exists for 1-th check, retry later table changefeed_dup_error_restart.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:26 CST 2025] <<<<<< run test case changefeed_dup_error_restart success! >>>>>> table mark.finish_mark_4 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:29 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b80eeae4-9a82-41d2-9019-77bf94fcb322 {"id":"b80eeae4-9a82-41d2-9019-77bf94fcb322","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752667} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1eba3fa b80eeae4-9a82-41d2-9019-77bf94fcb322 /tidb/cdc/default/default/upstream/7489360627009166534 {"id":7489360627009166534,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b80eeae4-9a82-41d2-9019-77bf94fcb322 {"id":"b80eeae4-9a82-41d2-9019-77bf94fcb322","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752667} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1eba3fa b80eeae4-9a82-41d2-9019-77bf94fcb322 /tidb/cdc/default/default/upstream/7489360627009166534 {"id":7489360627009166534,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b80eeae4-9a82-41d2-9019-77bf94fcb322 {"id":"b80eeae4-9a82-41d2-9019-77bf94fcb322","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752667} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1eba3fa b80eeae4-9a82-41d2-9019-77bf94fcb322 /tidb/cdc/default/default/upstream/7489360627009166534 {"id":7489360627009166534,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.27104.out cli changefeed create --start-ts=457114298649935873 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config /tmp/tidb_cdc_test/split_region/pulsar_test.toml table consistent_replicate_storage_s3.t2 not exists for 4-th check, retry later Create changefeed successfully! ID: b038bc94-f752-466d-b99c-3c8b913616ac Info: {"upstream_id":7489360627009166534,"namespace":"default","id":"b038bc94-f752-466d-b99c-3c8b913616ac","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:44:30.167554335+08:00","start_ts":457114298649935873,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114298649935873,"checkpoint_ts":457114298649935873,"checkpoint_time":"2025-04-04 15:44:25.138"} PASS =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:44:30 CST 2025] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> coverage: 2.4% of statements in github.com/pingcap/tiflow/... table mark.finish_mark_4 not exists for 3-th check, retry later + set +x table split_region.test1 not exists for 1-th check, retry later table consistent_replicate_storage_s3.t2 not exists for 5-th check, retry later table split_region.test1 exists table split_region.test2 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 24927 failed: Connection refused * Failed to connect to 127.0.0.1 port 24927: Connection refused * Closing connection 0 table consistent_replicate_storage_s3.t2 exists You are running an older version of MinIO released 4 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend table mark.finish_mark_4 not exists for 4-th check, retry later check diff successfully Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > Host: 127.0.0.1:24927 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 18330DD1D165891F < X-Xss-Protection: 1; mode=block < Date: Fri, 04 Apr 2025 07:44:35 GMT < { [226 bytes data] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... wait process cdc.test exit for 3-th time... table split_region.test2 exists check diff successfully table mark.finish_mark_4 not exists for 5-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:44:36 CST 2025] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1735117353.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_4 not exists for 6-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/safe_mode/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table mark.finish_mark_4 not exists for 7-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:39 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {UpstreamID:7489360454619399913 Namespace:default ID:2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.642459316 +0800 CST StartTs:457114289061494788 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c7cab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114289074864134} {CheckpointTs:457114289677533211 MinTableBarrierTs:457114302352457733 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/84d36262-36bb-4128-a8c7-33a76b320cd6 {"id":"84d36262-36bb-4128-a8c7-33a76b320cd6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752676} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1558ace 84d36262-36bb-4128-a8c7-33a76b320cd6 /tidb/cdc/default/default/changefeed/info/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"upstream-id":7489360454619399913,"namespace":"default","changefeed-id":"2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.642459316+08:00","start-ts":457114289061494788,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114289074864134} /tidb/cdc/default/default/changefeed/status/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"checkpoint-ts":457114289677533211,"min-table-barrier-ts":457114302352457733,"admin-job-type":0} /tidb/cdc/default/default/task/position/84d36262-36bb-4128-a8c7-33a76b320cd6/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360454619399913 {"id":7489360454619399913,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {UpstreamID:7489360454619399913 Namespace:default ID:2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.642459316 +0800 CST StartTs:457114289061494788 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c7cab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114289074864134} {CheckpointTs:457114289677533211 MinTableBarrierTs:457114302352457733 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/84d36262-36bb-4128-a8c7-33a76b320cd6 {"id":"84d36262-36bb-4128-a8c7-33a76b320cd6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752676} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1558ace 84d36262-36bb-4128-a8c7-33a76b320cd6 /tidb/cdc/default/default/changefeed/info/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"upstream-id":7489360454619399913,"namespace":"default","changefeed-id":"2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.642459316+08:00","start-ts":457114289061494788,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114289074864134} /tidb/cdc/default/default/changefeed/status/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"checkpoint-ts":457114289677533211,"min-table-barrier-ts":457114302352457733,"admin-job-type":0} /tidb/cdc/default/default/task/position/84d36262-36bb-4128-a8c7-33a76b320cd6/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360454619399913 {"id":7489360454619399913,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {UpstreamID:7489360454619399913 Namespace:default ID:2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:43:48.642459316 +0800 CST StartTs:457114289061494788 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003c7cab0 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114289074864134} {CheckpointTs:457114289677533211 MinTableBarrierTs:457114302352457733 AdminJobType:noop} span: {tableID:120, startKey:7480000000000000ff785f720000000000fa, endKey:7480000000000000ff785f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:121, startKey:7480000000000000ff795f720000000000fa, endKey:7480000000000000ff795f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:114, startKey:7480000000000000ff725f720000000000fa, endKey:7480000000000000ff725f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114302090051589, checkpointTs: 457114289677533211, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114289677533211, checkpointTs: 457114289677533211, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/84d36262-36bb-4128-a8c7-33a76b320cd6 {"id":"84d36262-36bb-4128-a8c7-33a76b320cd6","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752676} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc1558ace 84d36262-36bb-4128-a8c7-33a76b320cd6 /tidb/cdc/default/default/changefeed/info/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"upstream-id":7489360454619399913,"namespace":"default","changefeed-id":"2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:43:48.642459316+08:00","start-ts":457114289061494788,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114289074864134} /tidb/cdc/default/default/changefeed/status/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"checkpoint-ts":457114289677533211,"min-table-barrier-ts":457114302352457733,"admin-job-type":0} /tidb/cdc/default/default/task/position/84d36262-36bb-4128-a8c7-33a76b320cd6/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360454619399913 {"id":7489360454619399913,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x start tidb cluster in /tmp/tidb_cdc_test/multi_tables_ddl Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/safe_mode Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... table mark.finish_mark_4 not exists for 8-th check, retry later Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release table mark.finish_mark_4 not exists for 9-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_4 not exists for 10-th check, retry later Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_4 exists table mark.finish_mark_5 not exists for 1-th check, retry later check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360735404565085 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09bb680009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:44470, start at 2025-04-04 15:44:47.368217169 +0800 CST m=+2.388162877 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:47.376 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:47.372 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:47.372 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360735404565085 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09bb680009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:44470, start at 2025-04-04 15:44:47.368217169 +0800 CST m=+2.388162877 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:47.376 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:47.372 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:47.372 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360741287008882 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09bdb00005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-cdl5f-btd71-bm8n8, pid:44536, start at 2025-04-04 15:44:47.475689622 +0800 CST m=+2.399528609 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:47.484 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:47.468 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:47.468 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_5 not exists for 2-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360746141118996 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09cfc8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:20581, start at 2025-04-04 15:44:48.647168299 +0800 CST m=+2.201989994 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:48.655 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:48.626 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:48.626 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360746141118996 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09cfc8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:20581, start at 2025-04-04 15:44:48.647168299 +0800 CST m=+2.201989994 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:48.655 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:48.626 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:48.626 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360746313477922 TiDB Cluster ID. tikv_gc_leader_uuid 657ff09d2e8000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-kb04p-jr3kq-9r2lw, pid:20657, start at 2025-04-04 15:44:48.849884407 +0800 CST m=+2.310852582 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:46:48.858 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:44:48.826 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:34:48.826 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/safe_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/safe_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/safe_mode/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/safe_mode/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/safe_mode/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... [Fri Apr 4 15:44:51 CST 2025] <<<<<< START cdc server in multi_tables_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_tables_ddl.4586245864.out server --log-file /tmp/tidb_cdc_test/multi_tables_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_tables_ddl/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:44:51 CST 2025] <<<<<< run test case split_region success! >>>>>> table mark.finish_mark_5 not exists for 3-th check, retry later [Fri Apr 4 15:44:53 CST 2025] <<<<<< START cdc server in safe_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.safe_mode.2191521917.out server --log-file /tmp/tidb_cdc_test/safe_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/safe_mode/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_5 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:54 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b244a06f-a865-49d4-995b-b3602d3096b1 {"id":"b244a06f-a865-49d4-995b-b3602d3096b1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752691} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc24f93db b244a06f-a865-49d4-995b-b3602d3096b1 /tidb/cdc/default/default/upstream/7489360735404565085 {"id":7489360735404565085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b244a06f-a865-49d4-995b-b3602d3096b1 {"id":"b244a06f-a865-49d4-995b-b3602d3096b1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752691} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc24f93db b244a06f-a865-49d4-995b-b3602d3096b1 /tidb/cdc/default/default/upstream/7489360735404565085 {"id":7489360735404565085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b244a06f-a865-49d4-995b-b3602d3096b1 {"id":"b244a06f-a865-49d4-995b-b3602d3096b1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752691} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc24f93db b244a06f-a865-49d4-995b-b3602d3096b1 /tidb/cdc/default/default/upstream/7489360735404565085 {"id":7489360735404565085,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: test-normal Info: {"upstream_id":7489360735404565085,"namespace":"default","id":"test-normal","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:44:54.743152688+08:00","start_ts":457114305553235971,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t1","multi_tables_ddl_test.t2","multi_tables_ddl_test.t3","multi_tables_ddl_test.t4","multi_tables_ddl_test.t1_7","multi_tables_ddl_test.t2_7","multi_tables_ddl_test.finish_mark"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114305553235971,"checkpoint_ts":457114305553235971,"checkpoint_time":"2025-04-04 15:44:51.472"} Create changefeed successfully! ID: test-error-1 Info: {"upstream_id":7489360735404565085,"namespace":"default","id":"test-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:44:54.896608728+08:00","start_ts":457114305553235971,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t5","multi_tables_ddl_test.t6","multi_tables_ddl_test.t7","multi_tables_ddl_test.t8"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114305553235971,"checkpoint_ts":457114305553235971,"checkpoint_time":"2025-04-04 15:44:51.472"} Create changefeed successfully! ID: test-error-2 Info: {"upstream_id":7489360735404565085,"namespace":"default","id":"test-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:44:55.094924172+08:00","start_ts":457114305553235971,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t9","multi_tables_ddl_test.t10"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114305553235971,"checkpoint_ts":457114305553235971,"checkpoint_time":"2025-04-04 15:44:51.472"} table multi_tables_ddl_test.t55 not exists for 1-th check, retry later table mark.finish_mark_5 not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:44:56 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/366f3ed1-51f5-4472-bdbe-606b707eb2b2 {"id":"366f3ed1-51f5-4472-bdbe-606b707eb2b2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752693} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc25762e9 366f3ed1-51f5-4472-bdbe-606b707eb2b2 /tidb/cdc/default/default/upstream/7489360746141118996 {"id":7489360746141118996,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/366f3ed1-51f5-4472-bdbe-606b707eb2b2 {"id":"366f3ed1-51f5-4472-bdbe-606b707eb2b2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752693} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc25762e9 366f3ed1-51f5-4472-bdbe-606b707eb2b2 /tidb/cdc/default/default/upstream/7489360746141118996 {"id":7489360746141118996,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/366f3ed1-51f5-4472-bdbe-606b707eb2b2 {"id":"366f3ed1-51f5-4472-bdbe-606b707eb2b2","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752693} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc25762e9 366f3ed1-51f5-4472-bdbe-606b707eb2b2 /tidb/cdc/default/default/upstream/7489360746141118996 {"id":7489360746141118996,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.safe_mode.cli.21976.out cli changefeed create '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?safe-mode=true' Create changefeed successfully! ID: a01a183b-918c-4697-a685-9cda3e086da1 Info: {"upstream_id":7489360746141118996,"namespace":"default","id":"a01a183b-918c-4697-a685-9cda3e086da1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2025-04-04T15:44:56.566075919+08:00","start_ts":457114306850062342,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114306850062342,"checkpoint_ts":457114306850062342,"checkpoint_time":"2025-04-04 15:44:56.419"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table multi_tables_ddl_test.t55 not exists for 2-th check, retry later + set +x table mark.finish_mark_5 not exists for 6-th check, retry later table multi_tables_ddl_test.t55 not exists for 3-th check, retry later table mark.finish_mark_5 not exists for 7-th check, retry later check_redo_resolved_ts 2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e 457114307975446530 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e/meta global resolved ts 457114307333193730 not forward to 457114307975446530 run task failed 1-th time, retry later table mark.finish_mark_5 exists table mark.finish_mark not exists for 1-th check, retry later table multi_tables_ddl_test.t55 not exists for 4-th check, retry later check_redo_resolved_ts 2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e 457114307975446530 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e/meta global resolved ts 457114307857481731 not forward to 457114307975446530 run task failed 2-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table multi_tables_ddl_test.t55 not exists for 5-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } table mark.finish_mark exists check diff successfully [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit table multi_tables_ddl_test.t55 exists table multi_tables_ddl_test.t66 exists table multi_tables_ddl_test.t7 exists table multi_tables_ddl_test.t88 exists table multi_tables_ddl_test.finish_mark not exists for 1-th check, retry later [2025/04/04 15:45:06.921 +08:00] [DEBUG] [pd.rs:582] ["update latency statistics for raftstore-v1"] [tick=3362] [thread_id=97] [2025/04/04 15:45:06.922 +08:00] [DEBUG] [pool.rs:146] ["scheduling task inspect raftstore latency: RaftDisk"] [thread_id=97] [2025/04/04 15:45:06.922 +08:00] [DEBUG] [pd.rs:2055] ["executing task"] [task="inspect raftstore latency: RaftDisk"] [thread_id=30] [2025/04/04 15:45:06.922 +08:00] [DEBUG] [pool.rs:146] ["scheduling task compute slow score: id 3363, factor: RaftDisk, duration RaftstoreDuration { store_wait_duration: Some(21.266ยตs), store_process_duration: Some(12.81ยตs), store_write_duration: Some(13.716ยตs), store_commit_duration: Some(0ns), apply_wait_duration: None, apply_process_duration: None }"] [thread_id=89] [2025/04/04 15:45:06.922 +08:00] [DEBUG] [pd.rs:2055] ["executing task"] [task="compute slow score: id 3363, factor: RaftDisk, duration RaftstoreDuration { store_wait_duration: Some(21.266ยตs), store_process_duration: Some(12.81ยตs), store_write_duration: Some(13.716ยตs), store_commit_duration: Some(0ns), apply_wait_duration: None, apply_process_duration: None }"] [thread_id=30] [2025/04/04 15:45:06.927 +08:00] [DEBUG] [peer.rs:2639] ["renew lease"] [reason=tick] [peer_id=281] [region_id=280] [thread_id=92] [2025/04/04 15:45:06.927 +08:00] [DEBUG] [peer.rs:2639] ["renew lease"] [reason=tick] [peer_id=283] [region_id=282] [thread_id=91] [2025/04/04 15:45:06.927 +08:00] [DEBUG] [peer.rs:4287] ["request to get a read index"] [is_leader=true] [peer_id=281] [region_id=280] [request_id=6befca81-5d8b-486f-8ce9-fb52e23577e4] [thread_id=92] [2025/04/04 15:45:06.927 +08:00] [DEBUG] [peer.rs:3226] ["reset raft tick to Ordered"] [peer_id=281] [region_id=280] [thread_id=92] [2025/04/04 15:45:06.927 +08:00] [DEBUG] [peer.rs:2883] ["handle raft ready"] [peer_id=281] [region_id=280] [thread_id=92] [2025/04/04 15:45:06.927 +08:00] [DEBUG] [peer.rs:3489] ["handle reads with a read index"] [peer_id=281] [region_id=280] [request_id=6befca81-5d8b-486f-8ce9-fb52e23577e4] [thread_id=92] check_redo_resolved_ts 2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e 457114307975446530 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/2a1a0e22-6d09-4cd0-9b6e-9131f0e5700e/meta run task successfully wait process cdc.test exit for 1-th time... table multi_tables_ddl_test.finish_mark not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2025/04/04 15:45:09.188 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:45:09.189 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:45:09.189 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:45:09.190 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:45:09.193 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:45:09.193 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114302483267607] [resolvedTs=457114308905795587] [2025/04/04 15:45:09.194 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:45:09.194 +08:00] [INFO] [s3.go:423] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2025/04/04 15:45:09.194 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:45:09.194 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:45:09.194 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:45:09.199 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=5.499914ms] [2025/04/04 15:45:09.199 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:45:09.204 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.205 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:45:09.206 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:45:09.206 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:45:09.206 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:45:09.206 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:45:09.211 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:45:09.214 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:45:09.214 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] table multi_tables_ddl_test.finish_mark exists check table exists success + endpoints=http://127.0.0.1:2379 + changefeed_id=test-normal + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-normal -s + info='{ "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114306719776812, "checkpoint_time": "2025-04-04 15:44:55.922", "error": null }' + echo '{ "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114306719776812, "checkpoint_time": "2025-04-04 15:44:55.922", "error": null }' { "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 457114306719776812, "checkpoint_time": "2025-04-04 15:44:55.922", "error": null } ++ echo '{' '"upstream_id":' 7489360735404565085, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 457114306719776812, '"checkpoint_time":' '"2025-04-04' '15:44:55.922",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360735404565085, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 457114306719776812, '"checkpoint_time":' '"2025-04-04' '15:44:55.922",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-1 + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-1 -s + info='{ "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114310416007172, "checkpoint_time": "2025-04-04 15:45:10.022", "error": null }' + echo '{ "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114310416007172, "checkpoint_time": "2025-04-04 15:45:10.022", "error": null }' { "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 457114310416007172, "checkpoint_time": "2025-04-04 15:45:10.022", "error": null } ++ echo '{' '"upstream_id":' 7489360735404565085, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 457114310416007172, '"checkpoint_time":' '"2025-04-04' '15:45:10.022",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7489360735404565085, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 457114310416007172, '"checkpoint_time":' '"2025-04-04' '15:45:10.022",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-2 + expected_state=failed + error_msg=ErrSyncRenameTableFailed + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-2 -s + info='{ "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114305553235971, "checkpoint_time": "2025-04-04 15:44:51.472", "error": { "time": "2025-04-04T15:44:56.179830597+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' + echo '{ "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114305553235971, "checkpoint_time": "2025-04-04 15:44:51.472", "error": { "time": "2025-04-04T15:44:56.179830597+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' { "upstream_id": 7489360735404565085, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 457114305553235971, "checkpoint_time": "2025-04-04 15:44:51.472", "error": { "time": "2025-04-04T15:44:56.179830597+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSyncRenameTableFailed", "message": "[CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '138', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7489360735404565085, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114305553235971, '"checkpoint_time":' '"2025-04-04' '15:44:51.472",' '"error":' '{' '"time":' '"2025-04-04T15:44:56.179830597+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''138'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7489360735404565085, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 457114305553235971, '"checkpoint_time":' '"2025-04-04' '15:44:51.472",' '"error":' '{' '"time":' '"2025-04-04T15:44:56.179830597+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSyncRenameTableFailed",' '"message":' '"[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''138'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + message='[CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''138'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule.' + [[ ! [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '138', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule. =~ ErrSyncRenameTableFailed ]] check diff successfully [2025/04/04 15:45:11.262 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=2.06828453s] [2025/04/04 15:45:11.262 +08:00] [ERROR] [redo.go:272] ["ignore unsupported DDL"] [ddl="{\"StartTs\":457114302483267602,\"CommitTs\":457114302483267607,\"Query\":\"ALTER TABLE `consistent_replicate_storage_s3`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_s3`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"consistent_replicate_storage_s3\",\"Table\":\"t1\",\"TableID\":118,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:45:12 CST 2025] <<<<<< run test case multi_tables_ddl success! >>>>>> Exiting on signal: INTERRUPT [2025/04/04 15:45:12.776 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114308905795587] [2025/04/04 15:45:12.776 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114308905795587] [2025/04/04 15:45:12.877 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114308905795587] [2025/04/04 15:45:12.877 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114308905795587] [2025/04/04 15:45:13.379 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114308905795587] [2025/04/04 15:45:13.379 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:128, startKey:7480000000000000ff805f720000000000fa, endKey:7480000000000000ff805f730000000000fa}"] [checkpointTs=457114308905795587] [2025/04/04 15:45:13.479 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=457114308905795587] [2025/04/04 15:45:13.479 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:45:13.479 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:45:13.479 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:45:13.480 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] Apply redo log successfully The 1 times to try to start tidb cluster... check diff successfully [Fri Apr 4 15:45:14 CST 2025] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> count(*) 5000 Exiting on signal: INTERRUPT start tidb cluster in /tmp/tidb_cdc_test/multi_source Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 8185 0 --:--:-- --:--:-- --:--:-- 8185 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:45:12.225","puller_resolved_ts":"2025-04-04 15:45:08.125","last_synced_ts":"2025-04-04 15:42:58.275","now_ts":"2025-04-04 15:45:13.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:45:12.225","puller_resolved_ts":"2025-04-04' '15:45:08.125","last_synced_ts":"2025-04-04' '15:42:58.275","now_ts":"2025-04-04' '15:45:13.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 14881 8.9 0.0 15041736 148080 ? Sl 15:42 0:14 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 14934 6.4 0.0 14967236 142336 ? Sl 15:42 0:10 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":"2025-04-04T15:45:20.442531+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:45:20.442607+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:45:20.442727+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:45:20.442812+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:45:20.45357+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:20.454495+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:20.723444+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:45:20.723509+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:45:20.730711+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1743752722.2286115,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c73340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752722.228686,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752722.23065,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0032dae00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752722.2307465,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752722.4235144,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0040dfc00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752722.423569,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) {"level":"warn","ts":"2025-04-04T15:45:26.455339+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:26.45543+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:26.732532+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360905536188796 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c10d40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:27386, start at 2025-04-04 15:45:25.564595055 +0800 CST m=+2.291108764 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:25.572 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:25.557 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:25.557 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360905536188796 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c10d40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:27386, start at 2025-04-04 15:45:25.564595055 +0800 CST m=+2.291108764 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:25.572 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:25.557 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:25.557 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360893538010052 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c0f18000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:27452, start at 2025-04-04 15:45:25.48020307 +0800 CST m=+2.109219736 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:25.487 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:25.446 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:25.446 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_source/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_source/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/consistent_partition_table Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.28693.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release + set +x + tso='457114315653120001 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114315653120001 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:45:31 CST 2025] <<<<<< START cdc server in multi_source case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.2873228734.out server --log-file /tmp/tidb_cdc_test/multi_source/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_source/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 {"level":"warn","ts":1743752731.309379,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c73340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1743752731.5116227,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0040dfc00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) {"level":"warn","ts":"2025-04-04T15:45:32.455912+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:32.456738+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:32.732748+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:34 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47300369-6b63-4b0c-b238-56c4ed352756 {"id":"47300369-6b63-4b0c-b238-56c4ed352756","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e358e3 47300369-6b63-4b0c-b238-56c4ed352756 /tidb/cdc/default/default/upstream/7489360905536188796 {"id":7489360905536188796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47300369-6b63-4b0c-b238-56c4ed352756 {"id":"47300369-6b63-4b0c-b238-56c4ed352756","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e358e3 47300369-6b63-4b0c-b238-56c4ed352756 /tidb/cdc/default/default/upstream/7489360905536188796 {"id":7489360905536188796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47300369-6b63-4b0c-b238-56c4ed352756 {"id":"47300369-6b63-4b0c-b238-56c4ed352756","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752731} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc2e358e3 47300369-6b63-4b0c-b238-56c4ed352756 /tidb/cdc/default/default/upstream/7489360905536188796 {"id":7489360905536188796,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.28775.out cli changefeed create --start-ts=457114315653120001 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 870581ee-f1ef-43d8-ba53-cc4ccb68e7e7 Info: {"upstream_id":7489360905536188796,"namespace":"default","id":"870581ee-f1ef-43d8-ba53-cc4ccb68e7e7","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:45:34.944922593+08:00","start_ts":457114315653120001,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114315653120001,"checkpoint_ts":457114315653120001,"checkpoint_time":"2025-04-04 15:45:30.000"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360942706533331 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c9dac0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:18721, start at 2025-04-04 15:45:34.575501996 +0800 CST m=+2.863822323 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:34.583 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:34.571 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:34.571 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360942706533331 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c9dac0004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:18721, start at 2025-04-04 15:45:34.575501996 +0800 CST m=+2.863822323 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:34.583 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:34.571 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:34.571 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489360940211396886 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0c9d98000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-q97cb-x6ms7-n9j3z, pid:18786, start at 2025-04-04 15:45:34.603526166 +0800 CST m=+2.811823516 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:47:34.612 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:45:34.616 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:35:34.616 +0800 All versions after safe point can be accessed. (DO NOT EDIT) 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 [Fri Apr 4 15:45:38 CST 2025] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.2001320015.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:38.457048+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:38.458306+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:38.734725+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:45:41 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/adfb84cc-cd74-464b-859a-bea7ae6f24ab {"id":"adfb84cc-cd74-464b-859a-bea7ae6f24ab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752738} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc30c0ed4 adfb84cc-cd74-464b-859a-bea7ae6f24ab /tidb/cdc/default/default/upstream/7489360942706533331 {"id":7489360942706533331,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/adfb84cc-cd74-464b-859a-bea7ae6f24ab {"id":"adfb84cc-cd74-464b-859a-bea7ae6f24ab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752738} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc30c0ed4 adfb84cc-cd74-464b-859a-bea7ae6f24ab /tidb/cdc/default/default/upstream/7489360942706533331 {"id":7489360942706533331,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/adfb84cc-cd74-464b-859a-bea7ae6f24ab {"id":"adfb84cc-cd74-464b-859a-bea7ae6f24ab","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752738} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc30c0ed4 adfb84cc-cd74-464b-859a-bea7ae6f24ab /tidb/cdc/default/default/upstream/7489360942706533331 {"id":7489360942706533331,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 [2025/04/04 15:45:42.045 +08:00] [INFO] [main.go:86] ["running ddl test: 0 createDropSchemaDDL"] [2025/04/04 15:45:42.248 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:45:42.252 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:45:42.442 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:45:42.444 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:45:42.446 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:45:42.448 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:45:42.634 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:45:42.637 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 [2025/04/04 15:45:43.836 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:45:43.841 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:45:43.843 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:45:43.847 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:45:44.025 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:45:44.029 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:45:44.215 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:45:44.217 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:45:44.221 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:45:44.222 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] table safe_mode.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 4 15:45:40 CST 2025] <<<<<< run test case safe_mode success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:44.458707+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:44.459172+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:44.73668+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table partition_table2.t2 not exists for 1-th check, retry later [2025/04/04 15:45:45.431 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:45:45.438 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:45:45.778 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:45:45.786 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:45:45.792 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:45:45.801 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:45:47.000 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:45:47.003 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] table partition_table2.t2 not exists for 2-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 table partition_table2.t2 not exists for 3-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:50.459139+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:50.460667+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:50.738535+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table partition_table2.t2 not exists for 4-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:45:50 CST 2025] <<<<<< run test case kafka_big_messages success! >>>>>> [2025/04/04 15:45:52.555 +08:00] [INFO] [main.go:86] ["running ddl test: 1 truncateDDL"] [2025/04/04 15:45:52.758 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:45:52.768 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:45:52.944 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:45:52.945 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:45:52.968 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:45:52.970 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:45:53.131 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:45:53.171 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:45:53.324 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:45:53.324 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] table partition_table2.t2 not exists for 5-th check, retry later [2025/04/04 15:45:53.392 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:45:53.394 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:45:53.550 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:45:53.607 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 [2025/04/04 15:45:53.746 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:45:53.748 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:45:53.818 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:45:53.820 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:45:53.940 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:45:54.005 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:45:54.141 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:45:54.142 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:45:54.204 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:45:54.206 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:45:54.351 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:45:54.398 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 [2025/04/04 15:45:54.569 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:45:54.570 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:45:54.606 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:45:54.608 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:45:54.780 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:45:54.817 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:45:54.997 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:45:54.998 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:45:55.023 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:45:55.024 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:45:55.200 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:45:55.215 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:45:55.407 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:45:55.408 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:45:55.415 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:45:55.416 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] table partition_table2.t2 not exists for 6-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:55.443823+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:45:55.443883+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2025-04-04T15:45:55.443874+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:45:55.443919+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2025/04/04 15:45:55.626 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:45:55.630 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:55.724044+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2025-04-04T15:45:55.724079+0800","logger":"etcd-client","caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2025/04/04 15:45:55.832 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:45:55.833 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:45:55.838 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:45:55.840 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:45:56.033 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:45:56.062 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:45:56.241 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:45:56.247 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:45:56.281 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:45:56.282 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:45:55 CST 2025] <<<<<< run test case kafka_compression success! >>>>>> [2025/04/04 15:45:56.440 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:45:56.495 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] {"level":"warn","ts":"2025-04-04T15:45:56.459914+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:45:56.461239+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:45:56.739609+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} [2025/04/04 15:45:56.635 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:45:56.641 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:45:56.691 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:45:56.692 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:45:56.814 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:45:56.883 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:45:56.993 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:45:56.998 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:45:57.065 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:45:57.068 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] {"level":"warn","ts":1743752757.2302163,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002c73340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752757.2302775,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1743752757.231616,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0032dae00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1743752757.2316785,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2025/04/04 15:45:57.164 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:45:57.247 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] [2025/04/04 15:45:57.340 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] table partition_table2.t2 not exists for 7-th check, retry later {"level":"warn","ts":1743752757.4244697,"caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0040dfc00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1743752757.4245276,"caller":"v3@v3.5.12/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2025/04/04 15:45:57.349 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] [2025/04/04 15:45:57.430 +08:00] [INFO] [main.go:221] ["0 insert success: 2400"] [2025/04/04 15:45:57.436 +08:00] [INFO] [main.go:235] ["0 delete success: 1200"] [2025/04/04 15:45:57.518 +08:00] [INFO] [main.go:221] ["1 insert success: 2500"] table partition_table2.t2 not exists for 8-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:45:59 CST 2025] <<<<<< run test case kafka_messages success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 table partition_table2.t2 exists 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:46:02.460195+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000156e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2025-04-04T15:46:02.462565+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e80fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0{"level":"warn","ts":"2025-04-04T15:46:02.740873+0800","logger":"etcd-client","caller":"v3@v3.5.12/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002084e00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 28 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test [2025/04/04 15:46:03.439 +08:00] [INFO] [main.go:86] ["running ddl test: 2 addDropColumnDDL"] [2025/04/04 15:46:03.644 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:46:03.654 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:46:03.825 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:46:03.826 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:46:03.841 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:46:03.843 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:46:04.021 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:46:04.031 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] wait process cdc.test exit for 1-th time... [2025/04/04 15:46:04.202 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:46:04.203 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:46:04.212 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:46:04.214 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:46:04.384 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:46:04.389 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:04 CST 2025] <<<<<< run test case kafka_sink_error_resume success! >>>>>> [2025/04/04 15:46:04.670 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:46:04.673 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:46:04.676 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:46:04.677 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:46:04.937 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:46:04.940 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster [2025/04/04 15:46:05.124 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:46:05.125 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:46:05.126 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:46:05.128 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:46:05.311 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:46:05.314 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:46:05.498 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:46:05.501 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:46:05.507 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:46:05.508 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:46:05.681 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:46:05.689 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:46:05.880 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:46:05.882 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:46:05.882 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:46:05.883 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:46:06.066 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:46:06.069 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:46:06.250 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:46:06.251 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:46:06.269 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:46:06.272 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:46:06.434 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:46:06.457 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:46:06.782 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:46:06.783 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:46:06.805 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:46:06.807 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:46:06.961 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:46:06.989 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:46:07.140 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:46:07.140 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:46:07.172 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:46:07.174 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:46:07.324 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:46:07.356 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:46:07.505 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:46:07.506 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:46:07.541 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:46:07.543 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:07 CST 2025] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.2019820200.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server2 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2025/04/04 15:46:07.684 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:46:07.728 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:46:07.859 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:46:07.860 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:46:07.920 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:46:07.922 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] [2025/04/04 15:46:08.044 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] [2025/04/04 15:46:08.105 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:46:08.220 +08:00] [INFO] [main.go:235] ["0 delete success: 1200"] [2025/04/04 15:46:08.221 +08:00] [INFO] [main.go:221] ["0 insert success: 2400"] [2025/04/04 15:46:08.290 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] [2025/04/04 15:46:08.292 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] [2025/04/04 15:46:08.399 +08:00] [INFO] [main.go:221] ["0 insert success: 2500"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:08 CST 2025] <<<<<< run test case mq_sink_lost_callback success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:10 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [3971 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {UpstreamID:7489360942706533331 Namespace:default ID:478dc1fc-59a9-4449-b82f-1f76fb7c2c77 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:41.384179172 +0800 CST StartTs:457114318621114370 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00398b050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114318633959426} {CheckpointTs:457114326078849030 MinTableBarrierTs:457114326078849030 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/33c3fa8a-d7f0-4efa-85c1-6794fb96d61e {"id":"33c3fa8a-d7f0-4efa-85c1-6794fb96d61e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc30c1037 33c3fa8a-d7f0-4efa-85c1-6794fb96d61e /tidb/cdc/default/default/changefeed/info/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"upstream-id":7489360942706533331,"namespace":"default","changefeed-id":"478dc1fc-59a9-4449-b82f-1f76fb7c2c77","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:45:41.384179172+08:00","start-ts":457114318621114370,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114318633959426} /tidb/cdc/default/default/changefeed/status/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"checkpoint-ts":457114326078849030,"min-table-barrier-ts":457114326078849030,"admin-job-type":0} /tidb/cdc/default/default/task/position/33c3fa8a-d7f0-4efa-85c1-6794fb96d61e/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360942706533331 {"id":7489360942706533331,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {UpstreamID:7489360942706533331 Namespace:default ID:478dc1fc-59a9-4449-b82f-1f76fb7c2c77 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:41.384179172 +0800 CST StartTs:457114318621114370 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00398b050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114318633959426} {CheckpointTs:457114326078849030 MinTableBarrierTs:457114326078849030 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/33c3fa8a-d7f0-4efa-85c1-6794fb96d61e {"id":"33c3fa8a-d7f0-4efa-85c1-6794fb96d61e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc30c1037 33c3fa8a-d7f0-4efa-85c1-6794fb96d61e /tidb/cdc/default/default/changefeed/info/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"upstream-id":7489360942706533331,"namespace":"default","changefeed-id":"478dc1fc-59a9-4449-b82f-1f76fb7c2c77","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:45:41.384179172+08:00","start-ts":457114318621114370,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114318633959426} /tidb/cdc/default/default/changefeed/status/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"checkpoint-ts":457114326078849030,"min-table-barrier-ts":457114326078849030,"admin-job-type":0} /tidb/cdc/default/default/task/position/33c3fa8a-d7f0-4efa-85c1-6794fb96d61e/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} + grep -q 'failed to get info:' /tidb/cdc/default/default/upstream/7489360942706533331 {"id":7489360942706533331,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {UpstreamID:7489360942706533331 Namespace:default ID:478dc1fc-59a9-4449-b82f-1f76fb7c2c77 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2025-04-04 15:45:41.384179172 +0800 CST StartTs:457114318621114370 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00398b050 State:normal Error: Warning: CreatorVersion:v9.0.0-beta.1-10-ga306f4894 Epoch:457114318633959426} {CheckpointTs:457114326078849030 MinTableBarrierTs:457114326078849030 AdminJobType:noop} span: {tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating span: {tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}, resolvedTs: 457114326078849031, checkpointTs: 457114326078849030, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/33c3fa8a-d7f0-4efa-85c1-6794fb96d61e {"id":"33c3fa8a-d7f0-4efa-85c1-6794fb96d61e","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752768} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc30c1037 33c3fa8a-d7f0-4efa-85c1-6794fb96d61e /tidb/cdc/default/default/changefeed/info/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"upstream-id":7489360942706533331,"namespace":"default","changefeed-id":"478dc1fc-59a9-4449-b82f-1f76fb7c2c77","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2025-04-04T15:45:41.384179172+08:00","start-ts":457114318621114370,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"enable-table-monitor":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"content-compatible":false,"advance-timeout-in-sec":150,"send-bootstrap-interval-in-sec":120,"send-bootstrap-in-msg-count":10000,"send-bootstrap-to-all-partition":true,"send-all-bootstrap-at-start":false,"debezium-disable-schema":false,"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64","output-old-value":false,"output-handle-key":false},"open":{"output-old-value":true},"debezium":{"output-old-value":true}},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"synced-status":{"synced-check-interval":300,"checkpoint-interval":15},"sql-mode":""},"state":"normal","error":null,"warning":null,"creator-version":"v9.0.0-beta.1-10-ga306f4894","epoch":457114318633959426} /tidb/cdc/default/default/changefeed/status/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"checkpoint-ts":457114326078849030,"min-table-barrier-ts":457114326078849030,"admin-job-type":0} /tidb/cdc/default/default/task/position/33c3fa8a-d7f0-4efa-85c1-6794fb96d61e/478dc1fc-59a9-4449-b82f-1f76fb7c2c77 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7489360942706533331 {"id":7489360942706533331,"pd-endpoints":"","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:12 CST 2025] <<<<<< run test case mq_sink_dispatcher success! >>>>>> + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory [2025/04/04 15:46:14.316 +08:00] [INFO] [main.go:86] ["running ddl test: 3 addDropColumnDDL2"] [2025/04/04 15:46:14.516 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:46:14.523 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:46:14.722 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:46:14.724 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:46:14.738 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:46:14.740 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] [2025/04/04 15:46:14.904 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:46:14.934 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:46:15.080 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:46:15.080 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:46:15.120 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:46:15.122 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:46:15.263 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:46:15.306 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:46:15.440 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:46:15.441 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:46:15.486 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:46:15.488 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:46:15.633 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:46:15.685 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:46:15.818 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:46:15.819 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:46:15.876 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:46:15.878 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:46:16.010 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:46:16.078 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:46:16.191 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:46:16.192 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:46:16.268 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:46:16.270 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:46:16.366 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:46:16.464 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:46:16.546 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:46:16.547 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:46:16.666 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:46:16.668 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:46:16.733 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:46:16.875 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:46:16.934 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:46:16.934 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:46:17.052 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:46:17.054 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:46:17.112 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... [2025/04/04 15:46:17.225 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:46:17.284 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:46:17.285 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:46:17.406 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:46:17.407 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:46:17.454 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:46:17.577 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:46:17.623 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:46:17.623 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] [2025/04/04 15:46:17.745 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:46:17.746 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:46:17.790 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:46:17.916 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:46:17.959 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:46:17.959 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:46:18.083 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:46:18.085 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:46:18.126 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:17 CST 2025] <<<<<< run test case kafka_column_selector success! >>>>>> [2025/04/04 15:46:18.250 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:46:18.293 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:46:18.294 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:46:18.416 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:46:18.418 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] Verifying downstream PD is started... [2025/04/04 15:46:18.468 +08:00] [INFO] [main.go:221] ["0 insert success: 2300"] [2025/04/04 15:46:18.765 +08:00] [INFO] [main.go:221] ["1 insert success: 2300"] [2025/04/04 15:46:18.817 +08:00] [INFO] [main.go:235] ["0 delete success: 1200"] [2025/04/04 15:46:18.817 +08:00] [INFO] [main.go:221] ["0 insert success: 2400"] [2025/04/04 15:46:18.945 +08:00] [INFO] [main.go:221] ["1 insert success: 2400"] [2025/04/04 15:46:18.947 +08:00] [INFO] [main.go:235] ["1 delete success: 1200"] [2025/04/04 15:46:18.997 +08:00] [INFO] [main.go:221] ["0 insert success: 2500"] [2025/04/04 15:46:19.117 +08:00] [INFO] [main.go:221] ["1 insert success: 2500"] [2025/04/04 15:46:19.168 +08:00] [INFO] [main.go:235] ["0 delete success: 1300"] [2025/04/04 15:46:19.168 +08:00] [INFO] [main.go:221] ["0 insert success: 2600"] Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:21 CST 2025] <<<<<< run test case kafka_column_selector_avro success! >>>>>> Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:46:25.187 +08:00] [INFO] [main.go:86] ["running ddl test: 4 modifyColumnDDL"] [2025/04/04 15:46:25.408 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:46:25.410 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:46:25.585 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:46:25.586 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:46:25.587 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:46:25.588 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/debezium/run.sh using Sink-Type: mysql... <<================= =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:25 CST 2025] <<<<<< run test case lossy_ddl success! >>>>>> [2025/04/04 15:46:25.775 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:46:25.780 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:46:25.953 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:46:25.954 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:46:25.955 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:46:25.957 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:46:26.131 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:46:26.132 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:46:26.306 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:46:26.307 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:46:26.309 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:46:26.311 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:46:26.480 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:46:26.481 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:46:26.791 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:46:26.793 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:46:26.793 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:46:26.793 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:46:26.962 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] [2025/04/04 15:46:26.966 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:46:27.157 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:46:27.158 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:46:27.161 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:46:27.163 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361150936970107 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0faef4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:18192, start at 2025-04-04 15:46:24.842874702 +0800 CST m=+2.125847069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:24.851 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:24.829 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:24.829 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361150936970107 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0faef4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:18192, start at 2025-04-04 15:46:24.842874702 +0800 CST m=+2.125847069 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:24.851 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:24.829 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:24.829 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361161057964343 TiDB Cluster ID. tikv_gc_leader_uuid 657ff0fb2e40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:18251, start at 2025-04-04 15:46:25.083659795 +0800 CST m=+2.287638021 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:25.091 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:25.081 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:25.081 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2025/04/04 15:46:27.475 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:46:27.476 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] [2025/04/04 15:46:27.648 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:46:27.649 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:46:27.657 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:46:27.658 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:46:27.811 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:46:27.829 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:46:28.068 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:46:28.070 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:46:28.096 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:46:28.097 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:46:28.311 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] [2025/04/04 15:46:28.330 +08:00] [INFO] [main.go:221] ["0 insert success: 1500"] [2025/04/04 15:46:28.483 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:46:28.485 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] [2025/04/04 15:46:28.504 +08:00] [INFO] [main.go:235] ["0 delete success: 800"] [2025/04/04 15:46:28.505 +08:00] [INFO] [main.go:221] ["0 insert success: 1600"] [2025/04/04 15:46:28.893 +08:00] [INFO] [main.go:221] ["1 insert success: 1700"] [2025/04/04 15:46:28.905 +08:00] [INFO] [main.go:221] ["0 insert success: 1700"] [2025/04/04 15:46:29.075 +08:00] [INFO] [main.go:221] ["1 insert success: 1800"] [2025/04/04 15:46:29.076 +08:00] [INFO] [main.go:235] ["1 delete success: 900"] [2025/04/04 15:46:29.079 +08:00] [INFO] [main.go:235] ["0 delete success: 900"] [2025/04/04 15:46:29.080 +08:00] [INFO] [main.go:221] ["0 insert success: 1800"] + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19617.out cli tso query --pd=http://127.0.0.1:2379 [2025/04/04 15:46:29.246 +08:00] [INFO] [main.go:221] ["1 insert success: 1900"] [2025/04/04 15:46:29.253 +08:00] [INFO] [main.go:221] ["0 insert success: 1900"] [2025/04/04 15:46:29.591 +08:00] [INFO] [main.go:221] ["1 insert success: 2000"] [2025/04/04 15:46:29.593 +08:00] [INFO] [main.go:235] ["1 delete success: 1000"] [2025/04/04 15:46:29.597 +08:00] [INFO] [main.go:235] ["0 delete success: 1000"] [2025/04/04 15:46:29.598 +08:00] [INFO] [main.go:221] ["0 insert success: 2000"] [2025/04/04 15:46:29.797 +08:00] [INFO] [main.go:221] ["0 insert success: 2100"] [2025/04/04 15:46:29.801 +08:00] [INFO] [main.go:221] ["1 insert success: 2100"] [2025/04/04 15:46:29.965 +08:00] [INFO] [main.go:235] ["0 delete success: 1100"] [2025/04/04 15:46:29.966 +08:00] [INFO] [main.go:221] ["0 insert success: 2200"] [2025/04/04 15:46:29.981 +08:00] [INFO] [main.go:221] ["1 insert success: 2200"] [2025/04/04 15:46:29.983 +08:00] [INFO] [main.go:235] ["1 delete success: 1100"] ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7129 0 --:--:-- --:--:-- --:--:-- 7129 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:46:17.820","puller_resolved_ts":"2025-04-04 15:46:10.820","last_synced_ts":"2025-04-04 15:44:01.720","now_ts":"2025-04-04 15:46:18.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:46:17.820","puller_resolved_ts":"2025-04-04' '15:46:10.820","last_synced_ts":"2025-04-04' '15:44:01.720","now_ts":"2025-04-04' '15:46:18.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:46:17.820","puller_resolved_ts":"2025-04-04' '15:46:10.820","last_synced_ts":"2025-04-04' '15:44:01.720","now_ts":"2025-04-04' '15:46:18.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status_with_redo + mkdir -p /tmp/tidb_cdc_test/synced_status_with_redo + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status_with_redo shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory + set +x + tso='457114331258028033 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114331258028033 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114331258028033 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:46:30 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1964919651.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:30 CST 2025] <<<<<< run test case storage_csv_update success! >>>>>> chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status_with_redo Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:34 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cd6d9c6-9a5b-4825-a804-bf0623a732c7 {"id":"7cd6d9c6-9a5b-4825-a804-bf0623a732c7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752791} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3cb4be7 7cd6d9c6-9a5b-4825-a804-bf0623a732c7 /tidb/cdc/default/default/upstream/7489361150936970107 {"id":7489361150936970107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cd6d9c6-9a5b-4825-a804-bf0623a732c7 {"id":"7cd6d9c6-9a5b-4825-a804-bf0623a732c7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752791} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3cb4be7 7cd6d9c6-9a5b-4825-a804-bf0623a732c7 /tidb/cdc/default/default/upstream/7489361150936970107 {"id":7489361150936970107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7cd6d9c6-9a5b-4825-a804-bf0623a732c7 {"id":"7cd6d9c6-9a5b-4825-a804-bf0623a732c7","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752791} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc3cb4be7 7cd6d9c6-9a5b-4825-a804-bf0623a732c7 /tidb/cdc/default/default/upstream/7489361150936970107 {"id":7489361150936970107,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114331258028033 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19703.out cli changefeed create --start-ts=457114331258028033 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361150936970107,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:46:34.467135553+08:00","start_ts":457114331258028033,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114331258028033,"checkpoint_ts":457114331258028033,"checkpoint_time":"2025-04-04 15:46:29.528"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_update_pk_clustered/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:34 CST 2025] <<<<<< run test case csv_storage_update_pk_clustered success! >>>>>> Verifying downstream PD is started... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later [2025/04/04 15:46:36.097 +08:00] [INFO] [main.go:86] ["running ddl test: 5 addDropIndexDDL"] [2025/04/04 15:46:36.366 +08:00] [INFO] [main.go:221] ["0 insert success: 100"] [2025/04/04 15:46:36.381 +08:00] [INFO] [main.go:221] ["1 insert success: 100"] [2025/04/04 15:46:36.539 +08:00] [INFO] [main.go:221] ["0 insert success: 200"] [2025/04/04 15:46:36.558 +08:00] [INFO] [main.go:235] ["0 delete success: 100"] [2025/04/04 15:46:36.578 +08:00] [INFO] [main.go:221] ["1 insert success: 200"] [2025/04/04 15:46:36.581 +08:00] [INFO] [main.go:235] ["1 delete success: 100"] Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release [2025/04/04 15:46:36.856 +08:00] [INFO] [main.go:221] ["0 insert success: 300"] [2025/04/04 15:46:36.877 +08:00] [INFO] [main.go:221] ["1 insert success: 300"] [2025/04/04 15:46:37.027 +08:00] [INFO] [main.go:221] ["0 insert success: 400"] [2025/04/04 15:46:37.028 +08:00] [INFO] [main.go:235] ["0 delete success: 200"] [2025/04/04 15:46:37.045 +08:00] [INFO] [main.go:221] ["1 insert success: 400"] [2025/04/04 15:46:37.048 +08:00] [INFO] [main.go:235] ["1 delete success: 200"] [2025/04/04 15:46:37.213 +08:00] [INFO] [main.go:221] ["0 insert success: 500"] [2025/04/04 15:46:37.228 +08:00] [INFO] [main.go:221] ["1 insert success: 500"] [2025/04/04 15:46:37.545 +08:00] [INFO] [main.go:221] ["0 insert success: 600"] [2025/04/04 15:46:37.547 +08:00] [INFO] [main.go:235] ["0 delete success: 300"] [2025/04/04 15:46:37.568 +08:00] [INFO] [main.go:221] ["1 insert success: 600"] [2025/04/04 15:46:37.570 +08:00] [INFO] [main.go:235] ["1 delete success: 300"] [2025/04/04 15:46:37.859 +08:00] [INFO] [main.go:221] ["0 insert success: 700"] [2025/04/04 15:46:37.889 +08:00] [INFO] [main.go:221] ["1 insert success: 700"] [2025/04/04 15:46:38.122 +08:00] [INFO] [main.go:221] ["0 insert success: 800"] [2025/04/04 15:46:38.125 +08:00] [INFO] [main.go:235] ["0 delete success: 400"] [2025/04/04 15:46:38.155 +08:00] [INFO] [main.go:221] ["1 insert success: 800"] [2025/04/04 15:46:38.158 +08:00] [INFO] [main.go:235] ["1 delete success: 400"] [2025/04/04 15:46:38.514 +08:00] [INFO] [main.go:221] ["0 insert success: 900"] [2025/04/04 15:46:38.554 +08:00] [INFO] [main.go:221] ["1 insert success: 900"] Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t1 exists + sleep 5 [2025/04/04 15:46:38.882 +08:00] [INFO] [main.go:221] ["1 insert success: 1000"] [2025/04/04 15:46:38.885 +08:00] [INFO] [main.go:235] ["1 delete success: 500"] [2025/04/04 15:46:38.983 +08:00] [INFO] [main.go:221] ["0 insert success: 1000"] [2025/04/04 15:46:38.988 +08:00] [INFO] [main.go:235] ["0 delete success: 500"] [2025/04/04 15:46:39.318 +08:00] [INFO] [main.go:221] ["1 insert success: 1100"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_update_pk_nonclustered/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:46:38 CST 2025] <<<<<< run test case csv_storage_update_pk_nonclustered success! >>>>>> [2025/04/04 15:46:39.660 +08:00] [INFO] [main.go:221] ["0 insert success: 1100"] [2025/04/04 15:46:39.794 +08:00] [INFO] [main.go:221] ["1 insert success: 1200"] [2025/04/04 15:46:39.796 +08:00] [INFO] [main.go:235] ["1 delete success: 600"] [2025/04/04 15:46:40.015 +08:00] [INFO] [main.go:221] ["1 insert success: 1300"] [2025/04/04 15:46:40.066 +08:00] [INFO] [main.go:221] ["0 insert success: 1200"] [2025/04/04 15:46:40.069 +08:00] [INFO] [main.go:235] ["0 delete success: 600"] [2025/04/04 15:46:40.293 +08:00] [INFO] [main.go:221] ["1 insert success: 1400"] [2025/04/04 15:46:40.295 +08:00] [INFO] [main.go:235] ["1 delete success: 700"] [2025/04/04 15:46:40.413 +08:00] [INFO] [main.go:221] ["0 insert success: 1300"] [2025/04/04 15:46:40.576 +08:00] [INFO] [main.go:221] ["1 insert success: 1500"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2025/04/04 15:46:40.805 +08:00] [INFO] [main.go:221] ["0 insert success: 1400"] [2025/04/04 15:46:40.809 +08:00] [INFO] [main.go:235] ["0 delete success: 700"] [2025/04/04 15:46:40.881 +08:00] [INFO] [main.go:221] ["1 insert success: 1600"] [2025/04/04 15:46:40.886 +08:00] [INFO] [main.go:235] ["1 delete success: 800"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361224227515307 TiDB Cluster ID. tikv_gc_leader_uuid 657ff10a75c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:17887, start at 2025-04-04 15:46:40.734566409 +0800 CST m=+2.041448128 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:40.741 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:40.727 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:40.727 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361224227515307 TiDB Cluster ID. tikv_gc_leader_uuid 657ff10a75c0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:17887, start at 2025-04-04 15:46:40.734566409 +0800 CST m=+2.041448128 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:40.741 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:40.727 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:40.727 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361227610647502 TiDB Cluster ID. tikv_gc_leader_uuid 657ff10a8dc000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-b0k0m-r6x1k-4mcq1, pid:17946, start at 2025-04-04 15:46:40.868793465 +0800 CST m=+2.106724948 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:48:40.875 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:46:40.873 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:36:40.873 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status_with_redo/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17642 31.5 0.6 4881396 2540824 ? Sl 15:46 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 17643 23.0 0.6 4851700 2496588 ? Sl 15:46 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 17644 22.7 0.6 4847604 2472636 ? Sl 15:46 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 17646 31.0 0.6 4881908 2527020 ? Sl 15:46 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 11045 0 --:--:-- --:--:-- --:--:-- 11045 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:46:42.778","puller_resolved_ts":"2025-04-04 15:46:35.878","last_synced_ts":"2025-04-04 15:46:36.378","now_ts":"2025-04-04 15:46:43.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:46:42.778","puller_resolved_ts":"2025-04-04' '15:46:35.878","last_synced_ts":"2025-04-04' '15:46:36.378","now_ts":"2025-04-04' '15:46:43.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:46:42.778","puller_resolved_ts":"2025-04-04' '15:46:35.878","last_synced_ts":"2025-04-04' '15:46:36.378","now_ts":"2025-04-04' '15:46:43.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + cd /tmp/tidb_cdc_test/synced_status_with_redo + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19263.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114335412748289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114335412748289 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114335412748289 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status_with_redo --binary cdc.test [Fri Apr 4 15:46:46 CST 2025] <<<<<< START cdc server in synced_status_with_redo case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.1929719299.out server --log-file /tmp/tidb_cdc_test/synced_status_with_redo/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status_with_redo/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:46:49 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9fa05f50-df6f-4446-b2f5-c0a32cda5df1 {"id":"9fa05f50-df6f-4446-b2f5-c0a32cda5df1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752807} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc40db0e7 9fa05f50-df6f-4446-b2f5-c0a32cda5df1 /tidb/cdc/default/default/upstream/7489361224227515307 {"id":7489361224227515307,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9fa05f50-df6f-4446-b2f5-c0a32cda5df1 {"id":"9fa05f50-df6f-4446-b2f5-c0a32cda5df1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752807} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc40db0e7 9fa05f50-df6f-4446-b2f5-c0a32cda5df1 /tidb/cdc/default/default/upstream/7489361224227515307 {"id":7489361224227515307,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9fa05f50-df6f-4446-b2f5-c0a32cda5df1 {"id":"9fa05f50-df6f-4446-b2f5-c0a32cda5df1","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752807} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc40db0e7 9fa05f50-df6f-4446-b2f5-c0a32cda5df1 /tidb/cdc/default/default/upstream/7489361224227515307 {"id":7489361224227515307,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114335412748289 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status_with_redo.cli.19361.out cli changefeed create --start-ts=457114335412748289 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status_with_redo/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361224227515307,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:46:50.348762749+08:00","start_ts":457114335412748289,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114335412748289,"checkpoint_ts":457114335412748289,"checkpoint_time":"2025-04-04 15:46:45.377"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [2025/04/04 15:46:49.598 +08:00] [INFO] [main.go:78] ["runDDLTest take 1m7.553030445s"] table mark.finish_mark_0 exists table mark.finish_mark_1 exists table mark.finish_mark_2 exists table mark.finish_mark_3 exists table mark.finish_mark_4 exists table mark.finish_mark_5 not exists for 1-th check, retry later + set +x + sleep 20 table mark.finish_mark_5 exists table mark.finish_mark not exists for 1-th check, retry later table mark.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:46:55 CST 2025] <<<<<< run test case multi_source success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/vector/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/vector Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 15382 0 --:--:-- --:--:-- --:--:-- 15062 100 723 100 723 0 0 15382 0 --:--:-- --:--:-- --:--:-- 15062 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:46:45.377","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:47:11.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ jq .synced ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:46:45.377","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:47:11.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:46:45.377","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:47:11.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361373479061208 TiDB Cluster ID. tikv_gc_leader_uuid 657ff12e330000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:30430, start at 2025-04-04 15:47:17.372316116 +0800 CST m=+2.442843875 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:17.381 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:17.374 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:17.374 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361373479061208 TiDB Cluster ID. tikv_gc_leader_uuid 657ff12e330000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:30430, start at 2025-04-04 15:47:17.372316116 +0800 CST m=+2.442843875 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:17.381 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:17.374 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:17.374 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361383463409489 TiDB Cluster ID. tikv_gc_leader_uuid 657ff12de780006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-96b93-8j6sc-14nh9, pid:30513, start at 2025-04-04 15:47:17.065701023 +0800 CST m=+2.050321799 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:49:17.074 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:47:17.071 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:37:17.071 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/vector/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/vector/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/vector/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/vector/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/vector/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.31883.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114344928051201 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114344928051201 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 4 15:47:23 CST 2025] <<<<<< START cdc server in vector case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.3190831910.out server --log-file /tmp/tidb_cdc_test/vector/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/vector/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + check_logs /tmp/tidb_cdc_test/synced_status_with_redo ++ date + echo '[Fri Apr 4 15:47:23 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>>' [Fri Apr 4 15:47:23 CST 2025] <<<<<< run test case synced_status_with_redo success! >>>>>> + stop_tidb_cluster + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:47:26 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/119e2128-163f-4d79-adcd-45509f1fecac {"id":"119e2128-163f-4d79-adcd-45509f1fecac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752843} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc49761e3 119e2128-163f-4d79-adcd-45509f1fecac /tidb/cdc/default/default/upstream/7489361373479061208 {"id":7489361373479061208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/119e2128-163f-4d79-adcd-45509f1fecac {"id":"119e2128-163f-4d79-adcd-45509f1fecac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752843} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc49761e3 119e2128-163f-4d79-adcd-45509f1fecac /tidb/cdc/default/default/upstream/7489361373479061208 {"id":7489361373479061208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/119e2128-163f-4d79-adcd-45509f1fecac {"id":"119e2128-163f-4d79-adcd-45509f1fecac","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752843} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc49761e3 119e2128-163f-4d79-adcd-45509f1fecac /tidb/cdc/default/default/upstream/7489361373479061208 {"id":7489361373479061208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.vector.cli.31965.out cli changefeed create --start-ts=457114344928051201 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 9a1a5759-f5b7-42d3-95b9-89f2963c0bfc Info: {"upstream_id":7489361373479061208,"namespace":"default","id":"9a1a5759-f5b7-42d3-95b9-89f2963c0bfc","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2025-04-04T15:47:26.632667619+08:00","start_ts":457114344928051201,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114344928051201,"checkpoint_ts":457114344928051201,"checkpoint_time":"2025-04-04 15:47:21.675"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table test.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later table test.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 4 15:47:37 CST 2025] <<<<<< run test case vector success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/partition_table/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["partition_table.?*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" [2025/04/04 15:48:14.562 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2025/04/04 15:48:14.563 +08:00] [INFO] [apply.go:78] ["set memory limit"] [memoryLimit=10737418240] [2025/04/04 15:48:14.563 +08:00] [INFO] [apply.go:94] ["Start http pprof server"] [addr=127.0.0.1:6060] [2025/04/04 15:48:14.563 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2025/04/04 15:48:14.563 +08:00] [INFO] [redo.go:167] ["apply redo log starts"] [checkpointTs=457114326078849031] [resolvedTs=457114358100000773] [2025/04/04 15:48:14.564 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:48:14.564 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:48:14.564 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:48:14.565 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.164795ms] [2025/04/04 15:48:14.565 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.247331ms] [2025/04/04 15:48:14.569 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:48:14.572 +08:00] [INFO] [mysql.go:195] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [txn_cache.go:71] ["create new worker cache in conflict detector"] [cacheCount=16] [cacheSize=1024] [BlockStrategy=waitEmpty] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2025/04/04 15:48:14.573 +08:00] [INFO] [worker.go:90] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2025/04/04 15:48:14.573 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2025/04/04 15:48:14.573 +08:00] [WARN] [config.go:428] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2025/04/04 15:48:14.573 +08:00] [ERROR] [config.go:216] ["The TiDB source ID should never be set to 0. Please report it as a bug. The default value will be used: 1."] [tidbSourceID=0] [2025/04/04 15:48:14.577 +08:00] [INFO] [db_helper.go:161] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ALLOW_INVALID_DATES%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2025/04/04 15:48:14.579 +08:00] [WARN] [mysql_ddl_sink.go:291] ["please set `has-vector-type` to be true if a column is vector type when the downstream is not TiDB or TiDB version less than specify version"] [hasVectorType=false] [supportVectorVersion=8.4.0] [2025/04/04 15:48:14.579 +08:00] [INFO] [mysql_ddl_sink.go:122] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2025/04/04 15:48:14.579 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114326354100248,\"CommitTs\":457114326354100252,\"Query\":\"alter table t1 drop partition p1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":20,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:48:16.587 +08:00] [ERROR] [async_ddl.go:184] ["check async exec ddl failed"] [namespace=] [changefeed=] [error="Error 8121 (HY000): privilege check for 'Super' fail"] [2025/04/04 15:48:16.587 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114326354100252] [DDL="alter table t1 drop partition p1"] [2025/04/04 15:48:16.648 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=61.073515ms] [sql="alter table t1 drop partition p1"] [2025/04/04 15:48:16.648 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114326485172234,\"CommitTs\":457114326485172239,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:48:16.849 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114326485172239] [DDL="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`"] [2025/04/04 15:48:16.910 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=60.789449ms] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`"] [2025/04/04 15:48:16.910 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114326616506385,\"CommitTs\":457114326616506390,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:48:17.311 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114326616506390] [DDL="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`"] [2025/04/04 15:48:17.355 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=43.991542ms] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`"] [2025/04/04 15:48:17.355 +08:00] [WARN] [redo.go:287] ["apply DDL"] [ddl="{\"StartTs\":457114326865281034,\"CommitTs\":457114326865281038,\"Query\":\"ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"Schema\":\"partition_table\",\"Table\":\"t1\",\"TableID\":121,\"IsPartition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":64,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\",\"IsBootstrap\":false,\"BDRRole\":\"\",\"SQLMode\":0}"] [2025/04/04 15:48:17.758 +08:00] [INFO] [mysql_ddl_sink.go:225] ["Start exec DDL"] [namespace=] [changefeed=] [commitTs=457114326865281038] [DDL="ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))"] [2025/04/04 15:48:18.306 +08:00] [INFO] [mysql_ddl_sink.go:272] ["Exec DDL succeeded"] [namespace=] [changefeed=] [duration=547.960527ms] [sql="ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))"] [2025/04/04 15:48:18.306 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.306 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:117, startKey:7480000000000000ff755f720000000000fa, endKey:7480000000000000ff755f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.406 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.406 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:119, startKey:7480000000000000ff775f720000000000fa, endKey:7480000000000000ff775f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.506 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.506 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:115, startKey:7480000000000000ff735f720000000000fa, endKey:7480000000000000ff735f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.606 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.606 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:123, startKey:7480000000000000ff7b5f720000000000fa, endKey:7480000000000000ff7b5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.707 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.707 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:124, startKey:7480000000000000ff7c5f720000000000fa, endKey:7480000000000000ff7c5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.807 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.807 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:126, startKey:7480000000000000ff7e5f720000000000fa, endKey:7480000000000000ff7e5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.907 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:18.907 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:118, startKey:7480000000000000ff765f720000000000fa, endKey:7480000000000000ff765f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.007 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.007 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:116, startKey:7480000000000000ff745f720000000000fa, endKey:7480000000000000ff745f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.107 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.107 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:122, startKey:7480000000000000ff7a5f720000000000fa, endKey:7480000000000000ff7a5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.208 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.208 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:127, startKey:7480000000000000ff7f5f720000000000fa, endKey:7480000000000000ff7f5f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.308 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.308 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:129, startKey:7480000000000000ff815f720000000000fa, endKey:7480000000000000ff815f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.408 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span="{tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.408 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span="{tableID:133, startKey:7480000000000000ff855f720000000000fa, endKey:7480000000000000ff855f730000000000fa}"] [checkpointTs=457114358100000773] [2025/04/04 15:48:19.508 +08:00] [INFO] [redo.go:223] ["apply redo log finishes"] [appliedLogCount=47] [appliedDDLCount=4] [currentCheckpoint=457114358100000773] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2025/04/04 15:48:19.508 +08:00] [INFO] [worker.go:101] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] Apply redo log successfully check diff successfully [Fri Apr 4 15:48:19 CST 2025] <<<<<< run test case consistent_partition_table success! >>>>>> Database INFORMATION_SCHEMA METRICS_SCHEMA PERFORMANCE_SCHEMA mysql partition_table partition_table2 sys test =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:48:29 CST 2025] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:48:32 CST 2025] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:48:36 CST 2025] <<<<<< run test case multi_topics_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=mysql + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run mysql + '[' mysql '!=' storage ']' + return + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 4 15:48:39 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>>' [Fri Apr 4 15:48:39 CST 2025] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:48:42 CST 2025] <<<<<< run test case csv_storage_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:48:46 CST 2025] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Fri Apr 4 15:48:49 CST 2025] <<<<<< run test case csv_storage_partition_table success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 26777 0 --:--:-- --:--:-- --:--:-- 25821 100 723 100 723 0 0 26777 0 --:--:-- --:--:-- --:--:-- 25821 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:46:42.778","puller_resolved_ts":"2025-04-04 15:46:42.778","last_synced_ts":"2025-04-04 15:46:36.378","now_ts":"2025-04-04 15:48:53.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:46:42.778","puller_resolved_ts":"2025-04-04' '15:46:42.778","last_synced_ts":"2025-04-04' '15:46:36.378","now_ts":"2025-04-04' '15:48:53.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:46:42.778","puller_resolved_ts":"2025-04-04' '15:46:42.778","last_synced_ts":"2025-04-04' '15:46:36.378","now_ts":"2025-04-04' '15:48:53.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361892199824069 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1a08f4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:20909, start at 2025-04-04 15:49:14.452340119 +0800 CST m=+2.156193214 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:14.461 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:14.462 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:14.462 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361892199824069 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1a08f4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:20909, start at 2025-04-04 15:49:14.452340119 +0800 CST m=+2.156193214 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:14.461 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:14.462 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:14.462 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489361883157740637 TiDB Cluster ID. tikv_gc_leader_uuid 657ff1a0c840003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:20987, start at 2025-04-04 15:49:14.659431593 +0800 CST m=+2.279623249 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:51:14.667 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:49:14.657 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:39:14.657 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22225.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114375700348929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114375700348929 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114375700348929 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:49:20 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2226222264.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:49:23 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8d9c07f-aa04-49e8-8401-d0d0a24b93ed {"id":"c8d9c07f-aa04-49e8-8401-d0d0a24b93ed","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752960} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc663e0e6 c8d9c07f-aa04-49e8-8401-d0d0a24b93ed /tidb/cdc/default/default/upstream/7489361892199824069 {"id":7489361892199824069,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8d9c07f-aa04-49e8-8401-d0d0a24b93ed {"id":"c8d9c07f-aa04-49e8-8401-d0d0a24b93ed","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752960} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc663e0e6 c8d9c07f-aa04-49e8-8401-d0d0a24b93ed /tidb/cdc/default/default/upstream/7489361892199824069 {"id":7489361892199824069,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c8d9c07f-aa04-49e8-8401-d0d0a24b93ed {"id":"c8d9c07f-aa04-49e8-8401-d0d0a24b93ed","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743752960} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc663e0e6 c8d9c07f-aa04-49e8-8401-d0d0a24b93ed /tidb/cdc/default/default/upstream/7489361892199824069 {"id":7489361892199824069,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114375700348929 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22311.out cli changefeed create --start-ts=457114375700348929 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489361892199824069,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:49:23.973119721+08:00","start_ts":457114375700348929,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114375700348929,"checkpoint_ts":457114375700348929,"checkpoint_time":"2025-04-04 15:49:19.062"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20909 10.0 0.0 3332068 215872 ? Sl 15:49 0:02 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743752952283847425.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 20918 4.0 0.0 3405736 184436 ? Sl 15:49 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743752952293393665.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 20987 10.4 0.0 3110968 217040 ? Sl 15:49 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1743752952365518692.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 12789 0 --:--:-- --:--:-- --:--:-- 12789 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:49:31.362","puller_resolved_ts":"2025-04-04 15:49:25.412","last_synced_ts":"2025-04-04 15:49:25.412","now_ts":"2025-04-04 15:49:32.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:49:31.362","puller_resolved_ts":"2025-04-04' '15:49:25.412","last_synced_ts":"2025-04-04' '15:49:25.412","now_ts":"2025-04-04' '15:49:32.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:49:31.362","puller_resolved_ts":"2025-04-04' '15:49:25.412","last_synced_ts":"2025-04-04' '15:49:25.412","now_ts":"2025-04-04' '15:49:32.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 7366 0 --:--:-- --:--:-- --:--:-- 7129 100 221 100 221 0 0 7366 0 --:--:-- --:--:-- --:--:-- 7129 + synced_status='{"synced":true,"sink_checkpoint_ts":"2025-04-04 15:51:41.512","puller_resolved_ts":"2025-04-04 15:51:34.512","last_synced_ts":"2025-04-04 15:49:25.412","now_ts":"2025-04-04 15:51:42.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:51:41.512","puller_resolved_ts":"2025-04-04' '15:51:34.512","last_synced_ts":"2025-04-04' '15:49:25.412","now_ts":"2025-04-04' '15:51:42.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2025-04-04' '15:51:41.512","puller_resolved_ts":"2025-04-04' '15:51:34.512","last_synced_ts":"2025-04-04' '15:49:25.412","now_ts":"2025-04-04' '15:51:42.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Starting Downstream PD... Release Version: v9.0.0-beta.1.pre-91-g9d6bbb852 Edition: Community Git Commit Hash: 9d6bbb8529d9d2acb9cf324a65e94ece465f6346 Git Branch: master UTC Build Time: 2025-04-03 07:28:59 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 9.0.0-beta.1 Edition: Community Git Commit Hash: 3eb3439fd88f0db9de06ef92e845e80b32c9b49a Git Commit Branch: master UTC Build Time: 2025-04-02 02:57:52 Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Enable Features: memory-engine pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine trace-async-tasks openssl-vendored Profile: dist_release Starting Upstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v9.0.0-beta.1.pre-511-gd9fb3a5337 Edition: Community Git Commit Hash: d9fb3a53376157be49b5a7941e6e689a33c81eb0 Git Branch: master UTC Build Time: 2025-04-03 11:14:01 GoVersion: go1.23.8 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362612214117271 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2454f8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:23529, start at 2025-04-04 15:52:03.171407127 +0800 CST m=+2.097898686 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:54:03.179 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:52:03.184 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:42:03.184 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362612214117271 TiDB Cluster ID. tikv_gc_leader_uuid 657ff2454f8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:23529, start at 2025-04-04 15:52:03.171407127 +0800 CST m=+2.097898686 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:54:03.179 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:52:03.184 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:42:03.184 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 246 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 4 DDL Table Version. Do not delete. cluster_id 7489362602462161156 TiDB Cluster ID. tikv_gc_leader_uuid 657ff24568c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:pingcap-tiflow-pull-cdc-integration-test-3827-rkmnv-dl2gj-ffgv1, pid:23595, start at 2025-04-04 15:52:03.274533067 +0800 CST m=+2.121490545 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20250404-15:54:03.282 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20250404-15:52:03.285 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20250404-15:42:03.285 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v9.0.0-beta.1.pre-129-g8ecb79fcf3 Edition: Community Git Commit Hash: 8ecb79fcf30098e6786f4d0440ead2a879a3ad4a Git Branch: HEAD UTC Build Time: 2025-04-03 03:49:11 Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake Profile: RELWITHDEBINFO Compiler: clang++ 17.0.6 Raft Proxy Git Commit Hash: 11578499e429348400a384f1ca08a0200de99f0d Git Commit Branch: HEAD UTC Build Time: "" Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: external-jemalloc portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine openssl-vendored Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [18], vals: ["127.0.0.1:2379"] }, "memory-limit-ratio": MatchedArg { occurs: 1, indices: [6], vals: ["0.800000"] }, "config": MatchedArg { occurs: 1, indices: [10], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [16], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [12], vals: ["8ecb79fcf30098e6786f4d0440ead2a879a3ad4a"] }, "log-file": MatchedArg { occurs: 1, indices: [20], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [14], vals: ["v9.0.0-beta.1.pre-129-g8ecb79fcf3"] }, "addr": MatchedArg { occurs: 1, indices: [22], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 + is_tls=false + '[' false == true ']' ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24821.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='457114419930071041 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/...' + echo 457114419930071041 PASS coverage: 1.8% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=457114419930071041 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 4 15:52:09 CST 2025] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2485824860.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * connect to 127.0.0.1 port 8300 failed: Connection refused * Failed to connect to 127.0.0.1 port 8300: Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info --user ticdc:ticdc_secret -vsL * Trying 127.0.0.1... * TCP_NODELAY set * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Server auth using Basic with user 'ticdc' > GET /debug/info HTTP/1.1 > Host: 127.0.0.1:8300 > Authorization: Basic dGljZGM6dGljZGNfc2VjcmV0 > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 04 Apr 2025 07:52:12 GMT < Content-Length: 810 < Content-Type: text/plain; charset=utf-8 < { [810 bytes data] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/082b6d51-597d-4a2c-a99f-1829e3e2ac6c {"id":"082b6d51-597d-4a2c-a99f-1829e3e2ac6c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8f724e6 082b6d51-597d-4a2c-a99f-1829e3e2ac6c /tidb/cdc/default/default/upstream/7489362612214117271 {"id":7489362612214117271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/082b6d51-597d-4a2c-a99f-1829e3e2ac6c {"id":"082b6d51-597d-4a2c-a99f-1829e3e2ac6c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8f724e6 082b6d51-597d-4a2c-a99f-1829e3e2ac6c /tidb/cdc/default/default/upstream/7489362612214117271 {"id":7489362612214117271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/082b6d51-597d-4a2c-a99f-1829e3e2ac6c {"id":"082b6d51-597d-4a2c-a99f-1829e3e2ac6c","address":"127.0.0.1:8300","version":"v9.0.0-beta.1-10-ga306f4894","git-hash":"a306f48949e037965a7c1963f500f7e0abbfba51","deploy-path":"/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/bin/cdc.test","start-timestamp":1743753129} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/223195ffc8f724e6 082b6d51-597d-4a2c-a99f-1829e3e2ac6c /tidb/cdc/default/default/upstream/7489362612214117271 {"id":7489362612214117271,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=457114419930071041 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24917.out cli changefeed create --start-ts=457114419930071041 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/pull_cdc_integration_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7489362612214117271,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2025-04-04T15:52:12.686594046+08:00","start_ts":457114419930071041,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"enable_table_monitor":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64","output_old_value":false,"output_handle_key":false},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"content_compatible":false,"advance_timeout":150,"send_bootstrap_interval_in_sec":120,"send_bootstrap_in_msg_count":10000,"send_bootstrap_to_all_partition":true,"send-all-bootstrap-at-start":false,"debezium_disable_schema":false,"debezium":{"output_old_value":true},"open":{"output_old_value":true}},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v9.0.0-beta.1-10-ga306f4894","resolved_ts":457114419930071041,"checkpoint_ts":457114419930071041,"checkpoint_time":"2025-04-04 15:52:07.785"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 80333 0 --:--:-- --:--:-- --:--:-- 80333 + synced_status='{"synced":false,"sink_checkpoint_ts":"2025-04-04 15:52:07.785","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2025-04-04 15:52:34.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:52:07.785","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:52:34.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2025-04-04' '15:52:07.785","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2025-04-04' '15:52:34.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Fri Apr 4 15:52:45 CST 2025] <<<<<< run test case synced_status success! >>>>>>' [Fri Apr 4 15:52:45 CST 2025] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-pull_cdc_integration_test-3827/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS